Ask HN: Suggestions for spacewalk procedure writing?

96 points by jamesmontalvo3 13y ago ↗ HN

I work at NASA creating procedures for spacewalks (EVAs). Typically these procedures include 2 astronauts working outside the space station (called EV1 and EV2), plus a robotics operator inside, and other actions taken by mission control on the ground. All of this is happening in parallel. Sometimes EV1 may be working on a task where all the steps can be done independent of what EV2 is doing, while other times their steps are heavily intertwined. The same goes for robotic and mission control steps.

Currently we write these procedures in Microsoft Word. A procedure is basically a big table, with each column of the table being the steps for each entity. So EV1's steps are in one column, EV2's in another, and so on. If EV2 has a step that has to come after one of EV1's steps, then we make sure that the later step is lower on the page than the earlier step. All of this requires a lot of formatting in Word, and we spend too much time dealing with the formatting aspect. Most of these procedures are text with ordered and unordered lists, but there are also images and diagrams.

What I'm looking for:

- A better method for writing procedures with multiple people working in parallel

- Ability to easily see (and diff) the entire revision history

- Allow multiple editors. Not necessarily Google Docs-type multiple editors, though. Instead I imagine the procedure would reference individual documents as sub-procedures, and those could further reference more sub-procedures. So each editor could be revising smaller portions of the same spacewalk procedure, but not the same portion at the same time...so a lot like software development.

- Bonus if it can also help manage the location of hardware and tools as they're used throughout the spacewalk

Is there any such software in existence? If not, is anyone interested in creating such a thing? I'm not sure how much applicability outside spacewalks there this, but there has to be some.

36 comments

[ 3.5 ms ] story [ 79.4 ms ] thread

gee_totes 13y ago ↗

This may be a jumping off point, but possibly Markdown[0] syntax can help with your formatting issues. More advanced versions of the syntax also supports tables[1], so you could put in a column for EV1 and EV2.

Markdown files are stored in plain text, so diffing is easy.

Heck, looking at the first 3 requirements, you might be able to put EVA procedures all in the wiki section of a private Github repo.

[0]http://daringfireball.net/projects/markdown/syntax

[1]https://github.com/adam-p/markdown-here/wiki/Markdown-Cheats...

pfraze 13y ago ↗

Seems like you could use source control on plaintext files with a simple markup for specifying behaviors (doc embedding, syncing with a parallel process, etc). Then you write a program which reads the docs and guides movement along the procedure according to the markup.

andrewljohnson 13y ago ↗

I think it's important to separate the concept of what tool to use for editing vs. what tool you use for version control. You need a tool stack, maybe some glue code.

You could do it in HTML, then keep the HTML files in git so you can do version control. You can use any number of editors/WSIWYGs with HTML. You could then build custom tools to ease the workflow - are there any resources for this?

I thought first to say you could use Google Docs, but while it has a revision history and export to many editors, it does not do diffs well, and I'm not sure how well it handles your tabular format. I also doubt any particular online CMS designed for writing or editing will work particularly well, and meet all stakeholder needs.

itsprofitbaron 13y ago ↗

Reading your requirements specifically in relation to the procedures using “2 astronauts working outside the space station (called EV1 and EV2), plus a robotics operator inside, and other actions taken by mission control on the ground. All of this is happening in parallel. Sometimes EV1 may be working on a task where all the steps can be done independent of what EV2 is doing, while other times their steps are heavily intertwined. The same goes for robotic and mission control steps” it seems like you need a project management tool.

Whilst you’re currently using Microsoft Word for this, it would actually be much easier to do this using Excel (and create a Gantt Chart) OR you could use something like Asta Power Project[1] which is used in the Construction Industry as well as with Technology Organisations, Engineering, Manufacturing, Oil & Gas and Financial Services etc which rely on projects running in parallel.

The reason I recommended the Asta Power Project tool is because you can:

- Create accurate schedules easily using a drag/drop feature

- Have complete visibility over multiple projects in real-time

- Optimise your project resources

- Access powerful reporting tools

- ETC

I've actually used the Asta Power Project Software myself; it's pretty powerful and feature rich.

[1] http://www.astadev.com/products/asta-powerproject/

jamesmontalvo3 13y ago ↗

Thanks for this reference. I'll definitely take a look at Asta. As for Excel: yes there are some advantages to Excel, but it is not particularly easy to handle numbered lists within it. We need to be able to easily over voice communications explain where the crewmembers should go as the plans change on the fly, i.e. "EV1 must perform page 36 steps 15-27 prior to EV2 performing page 34 steps 1-10."

pytrin 13y ago ↗

You can use the 1st column for numbering the steps

trotsky 13y ago ↗

I agree that most of your requirements seem to have a lot in common with project planning tools that are commonly used to plan and manage software development with traditional methodologies. Microsoft project is the classic example, though industry trends seem to have made it a low priority for them.

All of these tools focus on creating gantt charts or similar, with independent subtasks that can be reliant on other tasks, broken into subtasks, group within barriers (milestones). They help enormously when you need to insert things or reorder stuff.

I've never seen one used to incorporate the actual detailed instructions for completing the task, generally because part of the task is figuring out how to do it. But I'd be surprised if you looked at a few different project management applications that there wouldn't be one or more that could be used that way. If nothing else you probably would be able to use the task description that's usually very short to include an arbitrary amount of text.

The biggest challenge might be in getting a report formatted to your liking at the end of the process with all the details nicely formatted. Most of these systems provide a great deal of extendability, so you might need to have some minimal development work done in terms of a custom report format. Or maybe not - these are enterprise systems that are usually loaded with special case features.

primitur 13y ago ↗

It's a long shot, but if it were me I'd put everything in a text MarkDown/Up language (train everyone on it), then have things laid out in a managed directory tree that is administered by way of a source repository tool (Git, &etc.)

The markup isn't necessarily a requirement, but some sort of text-based standard for how these documents are laid out would be important, imho.

For print-processing/missionpacks, I'd use latex. Its really the best possible way to solve the text formatting problem, especially if you use a markup. You could have your own markup dictionary/language too, which would fit right in with latex files.

So the whole policy group uses plain old text with smart formatting (so they are actually doing the work in-line) and doc generation through a latex pipe. Yup. Its the 70's all over again, and actually I'm becoming more and more aware of how non-ridiculous this is, while at the same time somehow being ridiculous. I wonder if there is a word for that ..

zdw 13y ago ↗

What you have is a graph (in the computing sense) of steps that happen along two parallel lines, with a bunch of steps that may or may not stop others from happening on the other line.

I'd look at things like a VCS's history graph, or video game skill trees to see how this could be visualized.

I'd break it down into text files with:

    - One file that covers all procedures
    - One file per procedure (or sub procedure)
    - One structured paragraph per step
    - A per-step list of tool/supply dependencies

This would be manageable in most VCS's, and could easily be broken out for work by multiple people. You could also have tools that automatically generates a list of all the tools used in a step/procedure, and then checks it against other procedures to avoid needing two of a tool when only one is available.

bcon 13y ago ↗

NASA has funded SBIR topics on automating EVA scheduling; TRACLabs (based in Clear Lake) has done some of this work, see http://traclabs.com/research/our-thought-leaders/ and particularly work by Debra Schreckenghost, Pete Bonasso, and Scott Bell. So NASA is developing automated scheduling tools and prototypes exist, they're just taking some time to reach your desk.

jamesmontalvo3 13y ago ↗

I've met Pete. Unfortunately their solutions don't handle the parallel nature of what we're doing. They have essentially written an Eclipse plug-in if I recall correctly.

bcon 13y ago ↗

They were using Eclipse as an interface for task creation and presentation; there's AI in the background ordering the tasks according to, eg, minimal procedure time. While their website is a little sparse, I think an email would prove useful.

Also, as I understand it, you're asking for solutions to the wrong problem. While any sort of project planner would be an improvement over scheduling in Word, the reason you don't use that already is a certain rigidity in how EVAs are planned. If that rigidity were overcome, you'd be able to use a simple project planner, and you'd also be able to use a much more advanced solution that ensures that no tasks collide.

raphman 13y ago ↗

In 2010 I saw a keynote by Ari Kristinn Jónsson [1] in which he explained how they designed and implemented semi-automated scheduling software for the Mars Exploration Rovers at NASA.

Could it be that there are people in NASA who already have solved your problem or understand the challenge better than the HN crowd?

[1] http://www.yourhost.is/images/stories/NordiCHI2010/accepted/...

sehugg 13y ago ↗

You might consider some sort of wiki software. MediaWiki (Wikipedia) and Confluence are two popular ones. They have the revision control and multiple editor support you're looking for. They can also be customized to add the bonus features you're looking for.

The downside is that you'll be administering a server and customizing it on an ad-hoc basis while you figure out how to organize the content and make it spit out the PDF format you're envisioning (that you might not be able to get perfect) and teaching everyone else how to use it properly. It might not even be appropriate if these procedures will be used once and then discarded.

The upside is that it's not Word.

jamesmontalvo3 13y ago ↗

We currently use Semantic MediaWiki for knowledge management. I've been considering creating an extension for it to handle procedure writing, but as you said I can't imagine getting it perfect. I feel like it would always be somewhat of a hack.

As for "these procedures will be used once and then discarded": that's not really an issue. Most procedures would not be used just once, since we'd reference a procedure for installing hardware if we ever needed to remove it. Procedures used to fix something will be reused if it breaks again.

bdunbar 13y ago ↗

It sounds like emacs org-mode would be what you're looking for.

- I use org-mode table feature to write checklists for system outages. Not as exiting as a space walk, granted.

- I don't use revision history but ... emacs. I bet it _has_ revision history.

- It's text. It can be edited by every editor you have heard of, and those you haven't.

- Underneath emacs is lisp. Assuming you've got a database of _some_ kind you can probably write a function to talk to it.

olalonde 13y ago ↗

Looks like you might be interested in UML Sequence Diagrams http://en.wikipedia.org/wiki/Sequence_diagram or Gantt charts http://en.wikipedia.org/wiki/Gantt_chart. There are multiple editors for those two but I don't know any that supports multiple people working in parallel.

kaybe 13y ago ↗

We're using LaTex with a repository.

Latex allows to split the document into many small files that can easily be worked on by different people. You can even split them further, and also compile just parts into pdf documents (\include to the rescue!).

A good repository (git works well multiplatform) can merge (automerge is very good) so it's not a problem if people work on the same file at the same time, as long as it's not exactly in the same place.

There are pretty pdf output files (which can be made to look like the normal word documents if it is needed). Graphs and Figures are handled automatically. Sometimes they don't do what one wants, but there is very extensive documentation out there.

The repository also takes care of the revision history.

Problems that I could see is that it's similar to writing html. Some people don't like this and might not want to use it. It's just a typesetting solution and will not do automatic time management (like checking that all is in the right order, or see what overlaps), but it is more powerful than Word.

(Seriously though, what is it with space agencies and Word? ESA/DLR sent us a mangled uncompilable latex template full of typos which said it came from "word-to-latex".)

threedaymonk 13y ago ↗

One technique that I saw recommended recently (here, possibly!) that works really well for version-controlled LaTeX documents is to write each sentence on a separate line.

Single line breaks don't affect the finished product, but separating sentences means that a change to one sentence never affects others around it.

dmckeon 13y ago ↗

You might consider reversing the approach - plan in terms of objectives and their dependencies, rather than in terms of sequential procedures and steps.

Unix environments have tools like tsort(1) and make(1) which process objective/dependency relationships, and many OS package managers which process dependency chains. (HN readers: no, I am not suggesting to just use Makefiles for spacewalks. :-) )

If you can build up a library of small procedures and dependency relationships among them, then express objectives in terms of a set of procedures, it should be possible (insert hand waving here) to generate (or regenerate) a reasonable sequence (or N parallel sequences) of steps from the underlying hierarchy.

Some questions:

How does complexity increase when evN > 2? IOW, what is big O(N)? (Yes, you only do 2 EVs now, but no doubt at some point N <= 2 will be become N > 2, and if you code for N then N>2 and N==2 will both be feasible.)

If parts of an EV plan must be radically altered during an EVA, how easy/feasible is it to generate a new plan, or do people fall back to other methods (skip/scrap portions of old plan, stop EVA & start over later, etc.)

dfc 13y ago ↗

Is this a joke?

jamesmontalvo3 13y ago ↗

As you said, we only do 2 EVs currently. However, we do have more than two entities working in parallel. For a hypothetical example, during a joint Space Shuttle and ISS EVA, there would be EV1 and EV2, but there might also be the Space Shuttle robotic arm and Space Station robotic arm (each run by at least one crewmember). Also, there may be a crewmember inside coordinating the EVA (reading procedures) who may have some tasks that need to be done to support the EVA, and there is commanding done from mission control. So that's 6 entities.

From an EVA planning perspective, we generalize that to three entities: EV1, EV2 and everyone else. That is, we have three columns in our Word document, and steps in the "everyone else" column will specify who is performing them, i.e. "Station Arm: translate to hardware pickup location"

jamesmontalvo3 13y ago ↗

Regarding your question on replanning: On the fly we won't generally have time to completely redo the procedure, and there's a lot of overhead to setup an EVA, so once outside the airlock we'll avoid scrapping the plan entirely and coming back inside unless there's no way to accomplish anything useful.

We almost always have extra tasks, which we don't consider part of the nominal procedure. These tasks can be used if for some reason we couldn't do part of the nominal procedure, or if we get the nominal procedure done vastly ahead of schedule and want to use the EVA time we already had book kept.

If, say, EV1 is 30 minutes ahead of EV2 and would otherwise just be waiting for EV2, we could send EV1 to do one of these extra tasks. In those cases normally we won't rewrite the procedure in anyway. A voice call will be made to the Flight Director (which everyone in mission control can here) saying something like "After EV1 completes step 45 he will have to wait for EV2 to complete step 56. We estimate that EV1 would be waiting 30 minutes. As such, we suggest he procede to Get-Ahead procedure 'Install Widget' on page 33"

Figs 13y ago ↗

Along the same sort of programmatic thinking, you could try writing the planned activities with a language that supports coroutines (like lua). Each actor can be modelled as a coroutine that is scheduled by a simple dispatch loop. The dispatch loop updates a clock counter, and the actors are written in terms of primitives like wait_for(condition, timeout) and perform_action(description, minimum_time, max_time) which yield in an internal loop based on the clock and some scenario settings. That way, you can test out proposed scenarios (what happens if it takes too long to finish activity X? Can we still finish Y if tool Z breaks? what happens if everyone takes the best timing? worst timing? random timings?).

weinzierl 13y ago ↗

TaskJuggler[1] is a free and open source software that can read task definitions from text files, schedule them and generate freely definable custom reports (including Gantt Charts). TaskJuggler allows you to partition the task definitions into several files. Together with a version control system (like git) this will allow you to:

        - define procedures with multiple resources working on tasks in parallel in a text file
        - generate a report describing the procedure
        - diff the entire revision history 
          (text diff has the most advanced diff and merge tools)
        - allow multiple editors ... a lot like in software development

For your bonus requirement: In TaskJuggler you can assign resources to tasks, and resources don't need to be people. I don't know if this solves the problem of which tool is where, but TaskJuggler is very flexible.

One thing that could turn out to be a problem is that TaskJuggler is meant for longer running projects (weeks to years). Maybe time resolution is not enough for your purpose.

[1] http://www.taskjuggler.org/ [2] http://en.wikipedia.org/wiki/TaskJuggler

bxbb 13y ago ↗

SharePoint is probably a good solution, considering your current workflow involving MS Word. Some features related to your problem:

- have revision support. A basic diff is available inside Word (Compare version).

- Breaking up the procedures as individual document can be done using outline view. Sub document can be created, imported from new files, or extracted from current master document.

- Co-authoring feature which enable multiple editors work on a same file, owner can review and accept/reject changes.

- Inventory management (I assume involving database) can be done using either Access or Excel.

What make SharePoint so nice is the Composites, which allow you to working with multiple datasource (Office files, SQL, Webservices). A handbook is available here: http://www.microsoft.com/en-us/download/details.aspx?id=3605...

Note: document outline doesn't require SharePoint (it's a built in feature of Word), co-authoring require either SharePoint or SkyDrive account to host the document.

dragonbonheur 13y ago ↗

I'd do that as a mind map in FreeMind, but I may be wrong. The SharedMind project allows for collaboration. http://code.google.com/p/sharedmind/

graeham 13y ago ↗

PERT [1] is a good technique, especially for parallel tasks, depending somewhat on how complex (how many) tasks need to be done. It is like a Gantt chart but IMHO better displays the data and dependencies and allows for additional information or referencing within the items. One of its downfalls can be when things change on the fly, but it should be possible for a software implementation to fix that.

I'm not aware of any software implementations, but I'd be surprised if there aren't any. Many applications need multiple tasks completed in convoluted series/ parallel. The automation industry does a lot of this kind of stuff for one, I am sure there are others. If not, it should be reasonably easy to make such a thing..

[1] http://en.wikipedia.org/wiki/Program_Evaluation_and_Review_T...

omegant 13y ago ↗

Airline pilot here, not the same of course, but I am used to sensible procedures that must be done on time, also I am aware of the importance of fast feedback and continuous evolution of procedures. Just some questions:

- How astronauts and mission control receive this procedures, written, computer, ipad, are google glass(or a similar HUD) a possibility to show the procedures live while working?

- Is it possible for them to make annotations, questions, have feedback on real time, or while preparing the mission? Could this be a desirable feature?

I don't really know of any tool that may be useful, I don't think that Microsoft project could really fit, it is too complicated for the task IMHO. I it should be something with this features (maybe not, Just guessing!):

- A version control like git that allows to save procedures, sub procedures, edits, comments, explanations or technical discussions that are not included in the "printed" version, informs and a tool to navigate all of them visually.

-User management: be able to have your own user, so as to have your edits in your account. Be able to follow another user or group of users.

- Word editor template a la google docs, that allows to see both EVA procedures, but with some excel like capabilities, like creating a time line(rows). Steps (cells) that can have a relationship with other cells from the same EVA(column), with the other EVA, with mission control and/or self properties (like tool to be used, comments, reports, etc..)

-Possibility to save sub-procedures to be used again, insert them in a different mission, also must have the possibility to be branched in a new mission to modify them. Also be reported, comment/discussion timeline, etc...

- Be multi-platform: laptop/PC/mac/linux, tablet, print format, and also HUD appliances(google glasses), (so the astronaut is able to scroll and read while on the vest).

-Also I suppose that multiple views of the same procedure must be possible, like simple, expanded, annotations, reports, versions, missions, incidents, warnings...

I don't know if a software of this kind could be created from parts of open source programs like open-office, git, etc.. or it must be created from cero.

Edit: typos.

7952 13y ago ↗

Microsoft OneNote. Multi user editing with easy linking between pages. Lovely to use and effortless to deploy. Just try it for ten minutes.

danbruc 13y ago ↗

The (enterprise) software developer's way.

  • define a XSD
  • define a XSLT to generate HTML or something similar

  • write your procedure in XML files and keep it under source control
  • transform the XML using the XSLT into HTML

The schema should allow to define tasks and dependencies and the transformation can then perform a topological sort and bring the tasks into the correct order. And you probably want a good XML editor with schema checking and autocompletion. An alternative to HTML would be LaTeX. You could also further process the output to generate a PDF.

brudgers 13y ago ↗

XML and CSS and a standard source management tool. The biggest problem with Word for something like this is that content and format are completely intermixed in the tool.

Word is designed to hide XML from the user and introduces significant complexity in order to do all sorts of things that this project doesn't need.

brudgers 13y ago ↗