Snakemake/Patterns - Revision history

Admin: /* Dealing with Paths */

2018-03-06T04:20:10Z

Dealing with Paths

@@ Line 207: / Line 207: @@
 * Use the PWD command to pass Docker an absolute path when mounting directories inside the container
 * Move anything the user might change to a .settings file - this will keep things as logical as possible
 ==Philosophy of Snakemake==

2018-03-03T03:33:03Z

The System - How to Grok Snakemake

← Older revision		Revision as of 03:33, 3 March 2018
Line 240:		Line 240:
	* Any variables inside the brackets must be python variables.		* Any variables inside the brackets must be python variables.
	* For example, {wildcard.name} means there is a Python variable named wildcard, that has an attribute called name. {output} means there is a Python variable named output. Etc.		* For example, {wildcard.name} means there is a Python variable named wildcard, that has an attribute called name. {output} means there is a Python variable named output. Etc.
			* If you need to access inputs or outputs by index, use the inputs variable like a Python variable: {inputs[0]} or {inputs[1]}
	* The shell and run directives have access to the same scope, and therefore if you can call the variable "{output}" in a shell block, you can call the variable "output" in the corresponding run block.		* The shell and run directives have access to the same scope, and therefore if you can call the variable "{output}" in a shell block, you can call the variable "output" in the corresponding run block.

2018-03-03T03:32:30Z

Philosophy in Practice

← Older revision		Revision as of 03:32, 3 March 2018
Line 225:		Line 225:
	* Likewise, the final master task aggregates the results of each subtask.		* Likewise, the final master task aggregates the results of each subtask.

	~~===Philosophy in Practice===~~		In practice, we do the following:

	In ~~terms of how this translates for Snakemake files~~, ~~here's how~~ the ~~organization looks~~:

	'''Subtask Snakemake Rules''':		'''Subtask Snakemake Rules''':

2018-03-03T03:32:10Z

Running Docker

@@ Line 136: / Line 136: @@
 </prE>
-==Running Docker==
+==Dealing with Paths==
-One of the issues is that Snakemake likes to think relative, but Docker needs absolute paths (particularly when specifying a directory to mount in the container as a volume).
+One of the issues I ran into with Snakemake was the fact that the input, output, and shell/run blocks expect the same filename, but each one expects it to be formatted differently. Furthermore, you may have the same filename but different paths.
-Solution (not tested): store PWD using os.get_cwd(), then use {PWD} in the Docker command
+This probelem is a big pain in the ass, and Snakemake does not help you solve it. This is also a BIG PROBLEM that clutters the Snakefile, and almost makes it worth looking for a different build tool. But we're stuck with Snakemake.
 ==Philosophy of Snakemake==

2018-03-02T08:20:54Z

Philosophy of Snakemake

← Older revision		Revision as of 08:20, 2 March 2018
Line 169:		Line 169:
	* Each subtask directory contains a Snakemake config file that defines defaults (which the user can change).		* Each subtask directory contains a Snakemake config file that defines defaults (which the user can change).
	* If there are multiple rules, rules specific to a rule file can also be defined in that file.		* If there are multiple rules, rules specific to a rule file can also be defined in that file.

			==The System - How to Grok Snakemake==

			To grok Snakemake, think about the Snakemake file as follows:
			* Each time you see a variable in brackets {} appearing in some shell command, think of those brackets as "opening" a Python scope.
			* Any variables inside the brackets must be python variables.
			* For example, {wildcard.name} means there is a Python variable named wildcard, that has an attribute called name. {output} means there is a Python variable named output. Etc.
			* The shell and run directives have access to the same scope, and therefore if you can call the variable "{output}" in a shell block, you can call the variable "output" in the corresponding run block.

	==Flags==		==Flags==

2018-03-02T08:17:08Z

Running Docker

@@ Line 112: / Line 112: @@
          """
 </pre>
 ==Running Docker==

2018-03-02T07:18:02Z

Executing commands in a temporary directory

2018-03-02T07:17:46Z

Dealing with complicated input/output lists

← Older revision		Revision as of 07:17, 2 March 2018
Line 68:		Line 68:
	data_dir = "data"		data_dir = "data"

	def MKOUT(data_dir, ~~fastq~~):		def MKOUT(data_dir, fastq_files):
	"""Returns a string that uses Snakemake's {} mini language"""		"""Returns a string that uses Snakemake's {} mini language"""
	~~return~~ os.path.join(data_dir,fastq)		result = []
			for fastq in fastq_files:
			p = os.path.join(data_dir,fastq)
			result.append(p)
			return result


	rule link_raw_data:		rule link_raw_data:
			"""imaginary scenario: complicated list of output files"""
	output:		output:
	MKOUT(data_dir, "{lib}_{rep}.fastq.gz"),		MKOUT(data_dir, ["{lib}_{rep}.fastq.gz", "{lib}_{rep}_small.fastq.gz"] ),
	params:		params:
	directory = data_dir,		directory = data_dir,
	~~shell_command = lib2data,~~
	message:		message:
	"Making link to raw data {output}."		"Making link to raw data {output}."
	shell:		shell:
	"""		"""
	(		do_stuff {params.directory}
	cd {params.directory}
	~~{params.shell_command}~~
	)
	"""		"""
	</pre>		</pre>

2018-03-02T07:15:20Z

Dealing with complicated input/output lists

← Older revision		Revision as of 07:15, 2 March 2018
Line 70:		Line 70:
	def MKOUT(data_dir, fastq):		def MKOUT(data_dir, fastq):
	"""Returns a string that uses Snakemake's {} mini language"""		"""Returns a string that uses Snakemake's {} mini language"""
	return data_dir ~~+ "/" +~~ fastq		return os.path.join(data_dir,fastq)

	rule link_raw_data:		rule link_raw_data:

2018-03-02T07:09:22Z

Dealing with complicated input/output lists

← Older revision		Revision as of 07:09, 2 March 2018
Line 66:		Line 66:

	<pre>		<pre>
			data_dir = "data"

	def MKOUT(data_dir, fastq):		def MKOUT(data_dir, fastq):
	"""Returns a string that uses Snakemake's {} mini language"""		"""Returns a string that uses Snakemake's {} mini language"""