Monitoring & DAG Visualization

SeqDesk provides real-time monitoring of pipeline runs through Nextflow’s weblog integration and trace file parsing.

Real-Time Monitoring

Weblog Integration

The primary monitoring method uses Nextflow’s weblog feature. When a pipeline runs, Nextflow sends HTTP events to SeqDesk’s weblog endpoint:


POST /api/pipelines/weblog?runId={runId}&token={secret}

Each event updates:

Run status — pending, queued, running, completed, failed
Step status — which Nextflow processes are running/completed
Progress — overall percentage (0–100)
Output tail — last ~100 lines of stdout/stderr

Trace File Fallback

If the weblog is unavailable, SeqDesk can monitor via the trace.txt file that Nextflow generates. A background monitor can be run:


# Continuous monitoring
npm run pipeline:monitor
 
# Single check (for cron jobs)
npm run pipeline:monitor:once

The trace monitor reads process execution data and updates run status accordingly.

Pipeline Run Statuses

Status	Description
`pending`	Run created, not yet started
`queued`	Submitted to SLURM, waiting for resources
`running`	Nextflow is executing processes
`completed`	All processes finished successfully
`failed`	One or more processes failed
`cancelled`	Run was manually cancelled

DAG Visualization

The pipeline definition file (definition.json) describes the workflow as a directed acyclic graph (DAG). SeqDesk renders this as an interactive visualization showing:

Each Nextflow process as a node
Dependencies between processes as edges
Current status of each process (color-coded)
Progress through the overall workflow

Process Steps

For the MAG pipeline, the main process steps are:

Step	Category	Description
SAMPLESHEET_CHECK	QC	Validate input samplesheet
FASTQC_RAW	QC	Quality check on raw reads
FASTP	Preprocessing	Read trimming and filtering
BOWTIE2_HOST_REMOVAL	Preprocessing	Remove host sequences
MEGAHIT	Assembly	Metagenome assembly
QUAST	QC	Assembly quality assessment
METABAT2	Binning	Genome binning
MAXBIN2	Binning	Alternative binning
DASTOOL	Binning	Bin refinement
CHECKM	QC	Bin completeness/contamination
GTDBTK	Annotation	Taxonomic classification
MULTIQC	Reporting	Aggregate QC report

Event Feed

Pipeline events are logged in the PipelineRunEvent table and shown in the monitoring UI:

Field	Description
Event Type	The type of event (process start, complete, error)
Process Name	Which Nextflow process
Status	Process status
Message	Human-readable description
Source	Where the event came from (weblog, trace, queue)
Timestamp	When the event occurred

Weblog Setup

To enable real-time monitoring:

Go to Admin → Pipeline Runtime
Set the Weblog URL to your SeqDesk instance’s public URL + /api/pipelines/weblog
Set a Weblog Secret for authentication
The URL is automatically included in the Nextflow command

The weblog URL format:


https://your-seqdesk.example.com/api/pipelines/weblog