Greenhouse demo¶
A generated greenhouse in Isaac Sim:
the smallest script that spawns GreenhouseCfg, which also renders the
greenhouse's picture for the docs; one that bends and cuts its lowest
trusses; and a pipe-rail robot harvesting them.

Contents: Requirements · Running it · Harvesting · Cutting and holding · Planning with cuRobo · Recording the video · Layout
Requirements¶
- uv installed
- Isaac Sim 6.2 compatible hardware
- Rust installed (this demo builds the scene generator from source)
Running it¶
uv run shot.py ../../docs/shots/greenhouse_demo/rows.json
opens Kit on the docs' picture: the greenhouse, the viewport at the
picture's size and camera. Navigate the viewport and press P, and the
terminal prints the camera, ready to paste into the shot. With an output
file after the shot, shot.py renders the picture headless instead.
uv run bend.py
opens Kit on one short row whose trusses below 1.8 m are rods. They settle
under their fruit, a capsule standing on the floor sweeps down each side of
the row through them, and once they are back at rest each is cut by its
stem and falls. --viz none runs it without a window and prints how far
the trusses were pushed, how close to where they settled they came back,
and how many fell.
uv run harvest.py
opens Kit on two short rows and the harvester at their heads, which cuts
every due truss below 1.8 m, where the trusses are rods, holds it, and
carries it to the crate on the lift, as the next section tells.
--paths picks the paths it works, every one by default, and
HEADLESS=1 uv run harvest.py --viz kit runs it without a window and
prints the tally once the last truss has fallen. --colliders draws every
collider the physics steps, translucent over the scene and kept in step
with the ones moved or resized while it runs, as
the pruning demo's does.
The first run of each generates its greenhouse and caches it; later runs
start straight up. The first harvest also fetches
cuRobo from GitHub and compiles its
kernels for the GPU it finds, into ~/.cache/curobo; no CUDA toolkit is
needed.
Harvesting¶
Warning
Work in progress. The harvest shows what the robot can do, not a finished controller: some trusses are not cut, and some are dropped short of the crate.
The robot has the layout of eternal.ag's Harvester and most robots of its kind: a trolley as wide as the heating pipes are spaced, a lift on it, a UR5e and a crate on the lift, and a cut-and-hold shear in its hand. The trolley does not roll here: it rides prismatic joints off a base fixed in the main path, across the row heads to a path, down it and up. The arm is MuJoCo Menagerie's UR5e, read out of its MJCF; the first run downloads it from the misina-lab-sim-assets dataset on Hugging Face, and later runs find it in the local cache.
The crate, harvester.Crate, is a solid-walled Euro stacking crate,
600 × 400 × 120 mm, standing on the lift behind the arm, turned across
it, under the folded shear and within the arm's reach. It is
drawn with the ribs down its walls, the rim round its top and the foot it
stacks on, and collides as five boxes: the floor, and each wall out to the
ribs' faces. The arm carries every truss it cuts to the crate and
lowers it in, at one of three spots along it; one it cannot carry there,
it lets go where it is.
The plan comes from the scene's own ground truth, not from a camera:
- What is due. A truss is due once its least ripe fruit is turning, by the stage the generator gives every fruit.
- The order. The robot crosses to each path in turn, cuts the due trusses on its left going down it and the ones on its right coming back, and folds the arm before every run of the trolley, so it never sweeps the crop wall.
- The approach. Each truss is cut by the stem, at the first joint of
its rod, which
Cutmarks within a centimetre: the blade crosses the peduncle half a centimetre past the joint, and the cut is made at it, so the whole truss falls. The blades' plane runs along the stem, a centimetre or so out from it, so the peduncle lies across the mouth, as square to it as the stem lets it be, and the shear slides past the stem rather than through it: a plane square to a peduncle that rises or droops off square to the stem would cross the stem a few centimetres from the cut. With the shear tilted up to 30° off level, the stem runs behind the blades and clear of the shear's head. It comes in along the row, or turned 25° about the stem toward the path, with the hook above the peduncle, so the truss's weight hangs the peduncle off the hook into the mouth, rather than dragging it over the hook's edge; the trolley stands behind the cut that way, so the arm reaches away from its base. cuRobo plans the arm to a stand-off 12 cm short of the cut, and the arm closes in on a straight line 35° below the blades, so the peduncle comes in through the middle of the open mouth's entrance, under the hook's claw, and rises into the hook's crook; along the blades, the claw, curling down past the crook, would sweep through it. The robot's weight is compensated, as a cobot's controller compensates its own, so the arm keeps to the line rather than sagging a centimetre below it, and holds still at its end until the mouth is at the cut. - The cut and the hold. The shear closes, its moving blade lifting the peduncle onto the hook, and cuts it where the blade reaches it, as the pruning demo's shear cuts a cane. Beside the blades, on the truss's side of the cut, a sprung finger riding the cutting blade presses the peduncle against a jaw beside the hook as the blade cuts it. A shear that closes on nothing opens, backs out and tries once more.
- The release. Held by its peduncle, the truss rides out on the line back to the stand-off and 20 cm across the path, clear of the crop wall, rising as it goes until the jaws are 42 cm above the crate's floor. The arm swings it round its base from there to over the crate, the mouth turning about the upright alone, so the truss hangs in the jaws as it did when cut, its weight pulling away from the mouth's entrance. Of three spots, the crate's middle and 12 cm along it either way, the truss goes to the one the arm reaches where it would come down furthest before a fruit of it meets a fruit already in the crate, the floor, or the rim where it reaches out over a wall, each fruit taken as the upright cylinder round it. The arm lowers it there, straight down at 15 cm/s, until it is 3 cm past that meeting, so the truss rests its weight on what lies under it, but with the jaws no lower than 10 cm above the rim: the wrist rides at about their height, and further down it meets the rim. A truss hangs about 15 cm below the jaws, so one lowered into the empty crate stops about 6 cm short of the floor. Where cuRobo finds the arm at the end of that line meeting the crate or its own links, it tries three quarters as far, then half, and failing those lets the truss go where it is. The shear opens and, where cuRobo finds the arm clear at the end of that line too, backs 6 cm off the truss the way it came in to the cut, so the peduncle passes out under the hook's claw. A truss let go short of the crate and still lying on the open finger is tipped off it. The forearm, the wrists, the shear and the crate collide with the trusses they meet, and the arm, the shear, the crate, the platform's deck and the trolley's chassis and mast with each other.
The tally counts the trusses due on the paths worked, how many the arm reaches, how many of those only by brushing a truss, how many it cuts, holds and lets go, how many end in the crate, and the seconds per truss cut. On the demo's two 4 m rows, 25 trusses are due on the three paths. The robot reaches and cuts every one, one only by brushing a truss, and 22 or 23 ride in the jaws to the crate and are lowered into it, runs differing by a truss or two. The rest are let go at the stand-off, where the arm cannot follow the line out across the path, or fall over the crate's rim as they settle. A truss lowered into the crate is let go with its lowest fruit 2 cm, in the median, over what lies under it. That is about 5.8 s of simulated time a cut and 280 s for the whole run, which takes about seven minutes on an RTX 5070 Ti.
Cutting and holding¶
The shear is the pruning demo's bypass shear, shear.py, which both demos
keep the same: a cutting blade swinging past a hook, colliding as convex
pieces along them, the cutting blade's in slices that stand at a rod while
the drawn blade closes on through it. The harvester's is smaller, a 6 cm
blade opening 35 degrees, and cuts and holds: 1.6 cm out along the pivot
from the hook, past the cutting blade, a rubber jaw stands beside the hook,
and a rubber finger rides the cutting blade on a joint of its own, held
against the blade by a spring.
A cane the pruning demo cuts is long and gives, and the moving blade carries it across the mouth onto the hook. A peduncle is short and stiff between its stem and the cut, so it is brought to the hook instead, a radius and a millimetre off its edge, halfway along the blade, where the cutting blade is widest: one the blade does not quite lift onto the hook is still between the blades as they meet, and is cut then.
The cut is a rule, and so is the hold. The finger meets the peduncle as
the blade does and stops there, while the blade closes on through and its
spring winds up, so the peduncle is pressed between the finger and the
jaw, with about 10 N once the shear is shut. The finger is one of the
robot's bodies the solver bending the trusses sees, as the blades are, so
the peduncle pushes it back as it would a real one. Friction there alone
would let a truss slide out of the jaws on its way to the crate, since that
solver meets the jaws as copies of the robot's bodies, posed anew from the arm
every step. So once the shear is shut, the segment of the cut peduncle
between the jaws is held where it lies in them and carried with the hand,
Shears.hold, the rest of the truss hanging from it. The shear opening
lets it go and swings the finger down and away from under the peduncle,
and the truss slides off the finger's slope and drops: nothing grips by
friction, so the hand's colliders, the shear's among them, slip, with a
friction coefficient of 0.25, 0.5 against a peduncle. One still lying on
the finger, where the shear was tilted up for the cut, is tipped off by
turning the blades down about the peduncle, as the pruning demo tips out a
piece of cane.
The hand's colliders are stiffer to the touch than the rest of the scene, too: 50 000 N/m, to Isaac Lab's default 2500. The solver bending the trusses lets a contact give in proportion to the load on it, and at the default a fruit borne up on the shear's head, its truss's weight on it, sinks a centimetre into it. At 100 000 N/m, the closing shear stalls on the peduncle.
The blade crosses the peduncle half a centimetre past its rod's first
joint, and Shears.cut makes the cut at the joint. Now and then the
peduncle shifts under the closing blade far enough that the cut lands a
centimetre or more past the joint, and the segment the jaws hold is the
next one, stretched back to the cut.
Planning with cuRobo¶
cuRobo plans every move of the arm: the fold before each run of the trolley, the reach to each stand-off and the swing to the crate. It plans the arm alone, from the platform out, while the trolley holds its station. What it collides with is what the simulation collides with:
- The robot.
harvester.pyholds the arm's colliders, Menagerie's capsules, and the box round each part of the shear as shapes: the head, and each blade with the jaw or the finger beside it. It writes the arm's into the URDF Isaac Sim imports, a capsule as its cylinder and a sphere on each end, beside the shear's own pieces, and hands cuRobo the spheres that cover each shape, spaced along a capsule no farther apart than its radius and grown to cover the dips between them, so a pose cuRobo finds clear is clear in the simulation too, the robot's collisions with itself included. The shoulder's are a centimetre smaller, clear of the deck it stands on. cuRobo plans with the shear held open and the finger against its blade. - The world.
planner.pyreads every collider of the running Newton model within the arm's reach, but the arm's own, and hands cuRobo the box round each where the simulation has it at that moment: the trolley's chassis and mast, the platform's deck, the crate, the rails, the floor, the stems, and the trusses and their fruit, bent or fallen. An upright stem's box reaches back to the middle of its row, a fin, so the arm keeps to the path's side of the stems and meets the crop only between two, where the trusses hang: a stand-off reached round the back of a stem is one cuRobo finds no way back out of. A stretch of stem left out by a cut leaves its fin out with it.
Bent trusses crowd the crop: a truss's fruit hang just below its cut and the next truss up a few centimeters above. A cut is planned clear of every truss first, but for the two segments the shear closes on, and clear of every stem but the 20 cm either side of the cut of the one it is made beside, which the shear slides along closer than the spheres cuRobo covers it in allow; the fold that follows leaves that stretch out too. Failing that, the arm may brush the truss it cuts, and failing that any truss, since the rods bend aside as they do round a picker's arm. It never touches the trolley, the rails, the floor or any other stretch of stem. The straight lines in, out and across the path are solved point by point, not planned, so they stay on the branch of the arm cuRobo reached the stand-off on, and within the joint range cuRobo plans in, so the fold that follows plans from where they end. Being solved point by point, a line keeps the arm off nothing, and the lines down into the crate and back off a truss let go there run by the crate's walls: each is made only where cuRobo finds a move to where it ends, the trusses left out. The swing to the crate may brush any truss, which leaves cuRobo fewer to keep the arm off and the mouth less to turn on the way. Where the arm already touches what cuRobo keeps it off, such as a stem, cuRobo finds no fold, and the arm folds on straight lines in joint space instead: out of the crop with the shoulder pan held, then round. Turned while still reaching into the crop, it would sweep along the crop wall into the stems.
Recording the video¶
./record.sh
films the harvester cutting Path_01's due trusses, the first three close up
and the rest from a camera riding along with the trolley, in Isaac Sim's RTX
renderer and headless, and writes
target/video/harvest.mp4 at the repo root, captioned and small enough to
upload, and a contact sheet beside it. It opens on the first truss, close
up, as the open shear pushes it from the side, brushes its lowest fruit and
lifts it, to show the trusses giving way to the arm, then cuts it. Its
storyboard -- the shots, the takes, those gestures, the pacing -- is the
top of video.py, and the run fails if a take's truss was not cut or a
gesture was not made.
docs/video/publish.sh target/video/harvest.mp4 examples/greenhouse_demo/README.md,
run from the repo root on the branch whose pull request changes the video,
uploads it and points the video line above at it.
Layout¶
| file | what it owns |
|---|---|
shot.py |
a still from a shot file in docs/shots/greenhouse_demo/: the greenhouse under a sky, or the harvester at work; see docs-shots |
bend.py |
a row with bent trusses under VBD, the capsules that sweep through them, and the cut |
harvest.py |
two rows and the harvester: the ground truth off the stage, the plan, the cuts, the holds and releases, the tally, and the gestures a video may have the arm make at the first truss |
harvester.py |
the robot: its dimensions, the UR5e read out of Menagerie's MJCF, its colliders, the shear on its flange, the URDF built from them and cuRobo's model of it, the actuators |
shear.py |
the cut-and-hold shear: its blades' geometry, their meshes and colliders, and Stroke, the blade closing through a peduncle, and the jaw and the sprung finger that hold the truss; the pruning demo's module, kept the same |
planner.py |
cuRobo: the world it plans in, read off the running simulation, the moves to joint positions, folded or over the crate, the reach to a cut or to any pose of the mouth, and its straight lines |
video.py, record.sh |
the video: Path_01 harvested, the first truss pushed, brushed and lifted before it is cut, filmed headless, and its encoding |
kinematics.py |
forward and inverse kinematics of the trolley and the arm, for the URDF, the stations and the straight line in; the pruning demo's module, kept the same |
examples/isaaclab_demo/DEVELOPMENT.md
covers the pinned Isaac Lab release, which this demo pins too.