Sysexy MIDI Librarian, technical documentation
================================================================

"Why do I have to keep reading these technical manuals?"

					-- Roger Waters,
					   Perfect Sense, Part I


I've tried to put most of the information about Sysexy that would be of use
to most users in the README file that accompanies the program, since most
people making music just want to make music.  This file is for those who
have further questions.


ABOUT SYSEXY

When I first adopted Linux as my platform for recording and sequencing, I
was actually quite pleased at all that was available -- Ardour for recording,
Muse and Seq24 for sequencing, Hydrogen for drum tracks, etc.  Frankly, I
first became involved in music during the 80's, at a time when most people
didn't have a computer at all, and typically you sequenced on a master
keyboard like an Ensoniq ESQ-1 or a Yamaha V50 using the built in sequencer
provided by the keyboard, you mixed on a physical mixer, and you recorded to
multitrack tape.  Post-production was done with rack effects.  Having the
computer provide anything at all was a plus for my generation of musicians.
To this day, I still like that sort of DAW-less approach sometimes.

But one thing that did underwhelm me on Linux was the availability (or lack)
of utilities for just doing sysex dumps.  There's the 'amidi' command, which
can do a generic dump (see following sections for descriptions of scenarios
where that won't be good enough), if you don't mind running CLI stuff during
your recording and sequencing sessions.  (At this point, we've already turned
off 90% of typical musicians.)  There was also a now-defunct Java application
called JSynthLib, which had a GUI, and could even handle dumps from a few of
the more finicky gear items I own...but I hate Java apps.

I was actually quite surprised.  Even in the 80's, on MS-DOS of all things,
you could still get a decent MIDI librarian program.  I didn't even know there
were platforms with lots of music software but no good sysex librarian
utilities.  I did know PERL, so I decided basically to go crazy and go down
the rabbit hole of writing my own MIDI librarian for Linux.  That created
Sysexy, so here we are.  I think I mostly created Sysexy to get around issues
previously experienced doing sysex dumps with my Oberheim OB-8, Siel DK600,
and Ensoniq ESQ-1 when using other methods.

Sysexy requires Perl/Tk, and a few other Perl modules that are commonly
shipped with just about all Linux distros.  One module it requires that is
probably not in your distro, MIDI::ALSA, you will probably have to obtain
through CPAN for now.  (There are instructions for that in the INSTALL
document.)


ANATOMY OF A SYSEX DUMP

When the MIDI specification was created in the 1980's, mostly it wanted to
be standard across manufacturers as much as possible.  The various keyboard
companies already had various proprietary systems for sequencing their own
equipment: Oberheim had an Oberheim sequencer, Roland had a Roland sequencer,
etc.  It was great as long as you were 100% brand loyal, but as soon as you
tried to connect machines from different companies, things got painful.  The
only prevalent non-proprietary interconnect for instruments that existed was
CV/gate, which was for all practical purposes monophonic, not polyphonic --
fine for Minimoogs and Roland SH-5's, but inadequate for anything you can
play chords on.  And whenever any company came up with a new scheme for
polyphonic interconnection, naturally it would belong to them and could be
used to hold whomever adopted it beholden to its creator company.  Naturally,
everyone was suspicious every time a keyboard company made a new system,
because usually that's exactly what they were always trying to do -- make a
new proprietary scheme they could rope their customers and competitors into.

But then Sequential Circuits did something unusual.  They created an
interconnect system in communication and participation with their competitors,
which would be completely open, non-proprietary, and would not make anyone a
slave of Sequential Circuits for adopting it.  It was called MIDI, and it's
actually sort of amazing how little it has changed in the last nearly forty
years.  Roland worked closely with them and was one of the first adopters,
and unlike CV/gate, it could transmit chords!  Roland quickly gave up on
their own proprietary DCB interconnect, which nobody ever used but Roland,
on the Jupiter-8 and Juno-60....  Their next synth, the Juno-106, had
excellent MIDI, at a time when MIDI had just been born.

But as much as MIDI wanted as much as possible to keep everything standard,
it is unavoidable that sometimes, one wants to send and receive data that
is not only brand-specific, but is of use only to the particular type of
machine that it came from -- so-called "system exclusive" data, the actual
sounds, settings, and sometimes even firmware of a particular model of a
particular brand of music/studio equipment.  One wants to be able to save
such things for archive and retrieval, knowing full well (and not caring)
that no other kind of equipment you ever see will have a use for it.  It
lets you keep more sounds, more sequences, more settings...more whatever
than will fit in the machine's own memory, and load such data as needed.
Basically, it's useful to be able to use MIDI for archival of stuff, even
if it's machine-specific stuff.

Thus, MIDI has a feature called "sysex," for system exclusive dumps of
data sets that are machine-specific.  Other devices may be able to
receive and store such dumps, but there is no expectation from MIDI or
anything else that a different device can actually understand what it's
storing.  For this reason, sysex dumps are also sometimes called "bulk"
dumps, since they're basically just a big block of bytes to whatever is
receiving it.  They don't have to be understood to be saved to disk raw.

Or at least that's the theory.  In practice, if you give a bunch of
commercial manufacturers the freedom to stick whatever they want in a big
block of bytes, and tell them, you can put whatever you want in here as
long as the receiving equipment doesn't have to know anything about it,
what really happens is they stick stuff in there that the _sending_
equipment _does_ have to know about.  And that's why you probably have
some equipment that can only do a sysex dump either to other equipment
from the same manufacturer, or to computer utilities that specifically
support that model of machine.

Why is that?  What makes some sysex dumps work with anything that can
receive a bulk dump at all, and others more...finicky?

In principal, a sysex dump starts with a header byte that is hexadecimal
0xF0 (decimal 240), and ends with a hex 0xF7 (decimal 247).  It ought to
be possible to watch for an F0 byte, assume everything after it is bulk
gibberish that needs to be saved to disk but not understood, followed by
an F7 that means the sender is done.  If it were a simple world, that's
all any sysex receiver would need to ever do.  If you have small generic
programs or MIDI hardware that can take a generic sysex dump, that's the
operating assumption that they make, which works much of the time.

But there are exceptions, and this is where Sysexy shines.  Sysexy is very
good at dealing with the devilish exceptions to this simple dump structure
that are seen in dumps from a lot of various kinds of equipment.


CHANNEL EMBEDDING

Remember when we said sysex dumps weren't supposed to contain anything
between the F0 at the beginning and the F7 at the end that needed to be
interpreted?  We lied.

Some equipment wants the MIDI channel to be embedded inside the dump.
Where in the dump?  That's peculiar to the particular brand and model of
equipment you happen to be getting a dump from.  The byte offset could
really be anywhere, but usually it's only a few bytes after the beginning
F0 byte, at some location the company picked out of a hat when they had an
employee lottery day.

It gets better.  MIDI channels only range from 1 to 16 (or 0 to 15 as they
are always handled internally by machines).  That's only a 4-bit range, a
nybble, as half a byte is typically called.  Since MIDI data is 7-bit (or
range: 0-127), that means a full channel range could be described without
using the top 3 bits in whatever 7-bit MIDI data byte it occured in.

It would make things easier if we could just assume that those 3 extra bits
will not get used for anything, and the byte in the dump where the MIDI
channel will get embedded is dedicated to just that.  Actually, with
Ensoniqs, that's the case:  There is a whole byte that gets embedded with
the MIDI channel, and the extra three bits at the top are zero.

Yamaha and Korg sometimes like to put other data in those three bits though
(waste not, want not, right?), so we can't just assume there's nothing
there.  Which means we're embedding nybbles here, not whole bytes.  Oh, fun.

It gets even better than that though.  One could hopefully presume that
because the high part of the nybble (the upper 3 bits) in a 7-bit number
would be inadequate to store the whole 0 to 15 range of possible MIDI
channels, no manufacturer would ever try to put the MIDI channel there.
Three bits only gives you a 0 to 7 (or 1 to 8) range.  That means they'd
have to make MIDI channels 9 through 16 unavailable if they tried to cram
the MIDI channel in the top half, right?

Well, I actually never have seen a machine that tries to embed the
channel in the 3 bits at the top...but I do have an Oberheim OB-8, which
can only use MIDI channels 1 through 8.  This means that it's not without
precedent for a synth manufacturer to forego half the available MIDI
channels just to make the channel selection fit into three bits, and the
fact that Oberheim didn't actually do this in a sysex dump doesn't mean
that nobody ever would.  So, just to make sure, Sysexy CAN handle all these
scenarios.  Sysexy can embed the MIDI channel at arbitrary byte offsets in
a dump.  It can do so in the low nybble (leaving data in the high three
bits undisturbed), and it can even do it in the high nybble (limiting you
to a channel selection of 1 through 8), in the unlikely event that there
actually is a piece of equipment out there that does what the OB-8 does
with MIDI channels, but even does it in their dump specification (which
the OB-8 doesn't do).  In other words, whatever lame-brain way your
equipment designer has decided to be hard to deal with using embedded
MIDI channels, Sysexy has a way to cope with it its gear database.

If you want to see some nify Perl regex, look at the channelize()
subroutine.  This function not only embeds the channel nybble, but will
even do so serially if the dump is made up of multiple concatenated
patch dumps strung together, into the right location in each dump in
the series.  Who says Perl isn't good for raw binary data?  :)


EMBEDDED PATCH NUMBERS

But, as they say in the television commercials, that's not all!  Things
get murkier when we consider dumping a single patch (or sound) by itself.
When the synthesizer receives the patch data, what does it do?  Some of
them will store it in their edit buffer, the limbo zone that sounds are
in while you're still adjusting them but you haven't saved it as a preset
yet.  This way, you send your patch to the synth, and then use the synth's
own menus to save the received data somewhere, as though it were a new
sound you'd just created by tweaking.

In this case, nothing special needs to happen inside the dump, and
everything is fine.  But there are some synths that expect to save an
incoming patch dump to memory immediately, in an actual preset memory.
How do they know where it will go?  Because you embedded that target
preset number into the dump, of course!  And you did it at a byte offset
inside the dump peculiar to that make and model of synth, which the
manufacturer picked out of a hat during a day when the employees had a
lottery (sound familiar?)!

This makes dumping of individual patches a source of their own kind of
pain for anyone trying to do this generically, with software or equipment
not made to talk to the particular unit you're trying to communicate with.
Add to this the possibility that in addition to embedding the patch number,
your machine may also expect the MIDI channel embedded as described earlier,
and you're talking about a whole lot of vendor/machine-specific formatting
inside a bytestream that was supposed to be possible for third parties to
treat as a "black box" that just needs to be archived for storage and
retrieved without the medium needing to do anything special.  Manufacturers
have interpreted "system exclusive" as meaning, "Surely you know everything
about our system, yes?  You know it exclusively, yes?"

Sysexy can deal with this also, to a point:  For synths whose program number
can fit into a single byte (range: 0 to 127, 7-bit MIDI), we provide the
ability to have an arbitrary byte that can contain the program number.  If
your instrument has multiple banks, or banks with more than 128 patches,
we simply cannot do this.  But to be fair, most librarians not made for a
specific piece of equipment can't deal with embedded patch numbers at all.
In the future, it's possible a later version of Sysexy will be able to
deal with embedding schemes for banks larger than 128 sounds, or multiple
patch banks, using whatever arbitrary embedding schemes there are out there.


MULTIPLE 0xF7's

But wait, there's more!  Remember when we said that a sysex dump is a stream
of bytes bumpered by a single 0xF0 at the beginning and a 0xF7 at the end?
Yeah, we lied about that too.  That would be if everyone followed the rules.

On some synthesizers, the type of dump used to send all the sounds at once
(known as a "bank" dump) doesn't really exist.  Instead, initiating such a
dump actually makes the synth send every patch in memory, one after another,
as consecutive single patch dumps, each one bumpered by its own individual
0xF0 and 0xF7 bytes before and after each patch.  They're all sent very
quickly with no pause in between, as a single transmission, but what you
get at the receiving end is essentially dozens of sysex dumps, all
concatenated together.  It's like the difference between a big block of
cheese, and slices at the grocery store that come individually wrapped and
stacked as though they were a single block of cheese.  (Ok, weird analogy.)

If you're receiving this with a generic utility or a hardware appliance
that expects 0xF7 terminates the dump, well, you just received the first
patch and then stopped listening.

Sysexy handles this with grace, especially if you've set it up with a
machine profile for the particular instrument you're using.  If not, it
can deal with this even in generic mode.


VERIFICATION

Sysexy can optionally make the instrument send again to make sure dumps
you've transmitted or received arrived correctly.  This setting is saved
per device, so it will only happen for gear that you find needs it.


PSEUDOBANKS

If you have a synthesizer that can't do bank dumps, but can do single patch
dumps with an embedded destination (as discussed earlier in SINGLE PATCH
DUMPS), Sysexy has a trick for you.  You can make your synth "fake" a bank
dump, in one of two different ways.

This bizarre feature was intended to help me with my Siel DK600, surely
one of the most awkward to use MIDI instruments ever made, at least for
purposes of integration with other equipment.  Siel gave the DK600 sysex
capability, but not for the reason that most synthesizers have it.

Siel made a companion product called the EX600, which was a keyboardless
tabletop module version of the DK600.  It had no real control panel of its
own, and the idea was supposed to be that you pair an EX600 module with a
DK600 keyboard, and the two would be able to sync patches between eachother
using an under-the-hood single-patch sysex transfer that happened while you
were using them.  Believe it or not, this sysex transfer even involved a
CC command!  (Talk about non-standard...)  There is no bank dump, since
Siel wasn't giving this to you so you could save your sounds on a computer;
they were doing it so your DK600 keyboard and your EX600 module could work
together.  Trying to use sysex on a DK600 for sound archival is basically
using the tool in a way the manufacturer never even thought about, even
though that would basically be the most normal thing to do with a sysex
dump on any other instrument.

However, the DK600 will accept single patch dumps with a destination
embedded (which is how the DK600 and EX600 plugged sounds into eachother
automatically when you use them together).  So...if we had a bunch of
single patch dumps, and we sent them serially, one after the other, very
quickly, couldn't we just send a "bank" dump of single patches, and fill
the receiving synth's memory with sounds?

We certainly could, and that's basically what Pseudobank does.  Sysexy
offers two different ways of doing this: "Pseudobank file" mode, and
"Pseudobank folder".  These two modes will let you send or receive many
single patch dumps to simulate a bank dump, for synths with limited
sysex capabilities that need this (or at the very least, for DK600/EX600
owners who need this).

In the Receive window, if you select "Pseudobank file" as your dump type
(which will only be possible if the synth you've selected is one that is
relevant to all this madness, like a DK600), Sysexy will do one of the
following depending on whether your synth supports dump request commands
or not:

1)  If your synth actually does have dump request commands, Sysexy will
    ask the instrument serially to send one patch after another to the
    computer, which will save it all in a single amalgamated file.  This
    file can later be sent from the Transmit window, with the Pseudobank
    button in that window turned on, to plug all the individual patches
    in the "bank" back into the synth.

2)  If it's like the DK600, and it doesn't even have dump request
    commands (ugh!), you will need to send from your instrument's control
    panel, one sound after another, manually, until you've sent them all.
    Painful, huh?  Here's the good part though:  Sysexy does not need for
    you to individually save on the computer end every time you send a
    patch from the keyboard.  Once you have a "Pseudobank file" receive
    going, Sysexy will keep listening, and will not stop listening until
    you click Done.  You can keep on sending patches from the DK600
    control panel until you've sent them all, with Sysexy listening the
    whole time, and when you're finished, click Done, and Sysexy will
    prompt you for a single file to amalgamate everything it just
    received into.  This file can be sent all at once from the Transmit
    window with the Pseudobank button on as though it were a bank dump.

Selecting "Pseudobank folder" does basically the same thing, except
instead of amalgamating all the incoming patches into a single file,
Sysexy will dump them individually into files with a numbered suffix,
after a base filename of your choosing.  It's recommended that you put
such a collection of single-patch, single-file dumps that belong together
as a "bank set" into a folder to keep them sorted, but it's not required,
and despite the dump type name "Pseudobank folder", Sysexy will not
actually make the folder for you, so you may want to do that beforehand.
(It does try very hard though to make sure that the set of files it's
about to write won't clobber anything already existing in the folder.)

All of this may seem like a very exotic feature, but I used to actually
have a crude bash script that worked around this with my DK600, and when
I wrote Sysexy, I decided that I wanted a more deluxe way of dealing with
this.  The DK600 is a warm, organic, punchy synthesizer with SSM filter
chips, and maybe this will help anyone else out there who has also
struggled to use it in a modern studio.  The synth is definitely worth it.


GEAR DATABASE EDITOR

Sysexy's gear database (typically stored in ~/.config/sysexy/geardb on
most systems) is just a bzip2-compressed text file generated by Perl's
Data::Dumper module, so I'm sure the more technically inclined users
will think nothing of loading that into a text editor of their choice
and hand-editing it.

But I'm writing this for musicians, and mostly they don't really want to
deal with things like that.  So, Sysexy has a sort of a wizard edit mode
to allow you to add, edit, and delete instruments from your stored gear
description database, which will allow you to add machines that I didn't
have, or one that haven't found their way into Sysexy yet, provided you
have the MIDI specification data that will be necessary to do that.

And that last part is the reason why this is in the TECH manual and
not the regular README manual:  Even with a wizard to edit it, creating
an instrument profile requires diving into hexadecimal command strings,
looking at headers of dump files in raw binary, etc.  In other words,
you will be looking at the appendix of your synth manual reading the
parts nobody ever looks at, at the very least, if not doing a whole
lot more than that, by the time you're done with this.

If you do happen to create a new instrument profile, please feel free to
share your settings, so it can be included in future versions of Sysexy.

But for now, here's some help with the wizard questionnaire:

Question 1, Instrument name:
Put whatever you'd like here, but you can't use brackets.  Also, don't
distinguish too much, as in between a synth keyboard and its own rackmount
version of the same instrument, unless you really know that is a different
kind of sysex dump.

Question 2, Dump type:
Sysexy has special handling and understanding of "Bank" (a dump of all
patches), and "Patch" (a dump of a single patch, possibly with a patch
number embedded), but beyond that, you can create dump types with whatever
names you want as needed, and as many types per instrument as needed.
In other words, nothing in the program will stop you from creating an
"Elephant Tumor" dump type if you really think your instrument can send
something you'd want to call that.  Aside from "Bank" and "Patch",
dump type names are just names.

Question 3, Tag:
Here you enter a unique tag for this type of dump which has to be 16
characters or less and not already occuring anywhere in the database.
It may contain spaces and will be visible in the message window.
Basically something short and sweet, like "ESQ1 bank".

Question 4, Request command:
This is where the fun starts.  By command, we're talking about a string
of hexadecimal, not something you'd type at a prompt.  Go to the back of
your instrument's manual, probably in a section called MIDI Specification.
Starting in the 90's, many manufacturers started providing a glossed-over,
one page long specification sheet in their manuals that doesn't actually
tell you much, so you may have to look elsewhere for a companion pamphlet
that has the real info you need.  (Usually it will turn up on Google.)  If
the MIDI specification you have is only one page long and it doesn't have
any hexadecimal in it, you've got the glossed over version, and you need
to look for the "real" MIDI specification for your hardware, which should
be out there somewhere.  When you find it, it may need channel numbers,
patch numbers, etc. embedded at various points, and that can be described
here using notation that's described in the Sysexy's wizard.  Different
companies produce documents for explaining this with various degrees of
human intelligibility.  Ensoniq couldn't be easier.  Yamaha and Korg are
fair.  Lexicon MIDI specs were written by aliens from space.  Your
mileage may vary.

Question 5, Header:
Now that you've gone digging through product documentation not meant for
the normal consumer, you will now need to run a binary editor and look at
data not meant for humans.  You need to find the bytes at the beginning of
a dump of this type that distinguish it from other types of files, but
without including bytes that are particular to the data in this specific
actual file (i.e., the actual sound data from this particular dump).  So
you're trying to find the fingerprint of this dump type, so Sysexy can
recognize it on disk even if the filename doesn't tell you what it is.
If this seems like a pain, remember, this is something you do for other
users so it will be automatic for them.  You, the machine profile creator,
will have to do a little work here.  As described in the wizard, there is
notation you can use in your header to denote embedded MIDI channels,
patch numbers, and other things like that, which Sysexy will omit in
matching the fingerprint you enter to files it sees on disk.  In other
words, this will make Sysexy not care if the file it sees doesn't have
some particular literal embedded number at that location.

Question 6, Header byte length:
How many bytes of real header data does the fingerprint you just entered
describe?  Don't count characters in your string.  We're talking about
actual bytes in the sysex file here, which is usually not the same.

Question 7, Dump byte size:
Most synthesizers produce dumps that are the same number of bytes long,
per type of dump, every time they do a dump of that given type.  If yours
does also, enter here how many bytes long this type of dump is, so Sysexy
will know if it's incomplete just from number of bytes of data received.
Enter "0" here if the size may vary, or in a pinch if you really can't
find out yet and you don't know.  Please try and find out for certain
though before you distribute a machine profile to others, because Sysexy's
error detection works much better when it knows the expected bytesize.

Question 8, 0xF7 total:
Usually the answer to this will be "1", but there are synthesizers whose
bank dumps are actually concatenations of lots of patch dumps strung
together.  In cases like this, enter here how many 0xF7 terminator bytes
there really are in this type of dump, which for a synth that behaves as
described above, will probably end up being the same as the number of
patch presets in its memory.

Question 9, Patch byte ("pbyte"):
The wizard will only display this question if you selected dump type
"Patch", and it's asking if there is a byte in the dump that needs to
contain the program number (or patch number).  Leave it at zero if
your instrument doesn't do that, or set it to the byte offset with
the first byte considered location "0" if it does need that.  Typically
this will be the number of bytes after the 0xF0 at the beginning.

Question 10, Highest patch number ("pmax"):
Sysexy currently accepts a range of up to 128 patches in a bank for synths
that require the embedded patch number, but it can be lower.  If your synth
does require patch number embedding, you can specify a high patch number,
especially if it's something lower than 127.  This question will only
appear if you gave a non-zero answer to question 9.

Question 11, Tested?
Distributing/sharing your Sysexy machine profiles with others (and with me!)
is encouraged, because they can be included in future versions of the program
so that Sysexy will directly support more gear by default.  But this question
encourages you to actually try your profiles on the real device, since I have
found MIDI information in some instrument manuals that is just totally wrong.
Did you actually try this profile on the actual instrument?  And it worked?

Question 12, Advice:
Entering information here is optional and usually not necessary, but here you
can enter some notes that will appear in the "Advice" dialog in the Receive
and Transmit windows.  You can offer advice to other owners of the same gear
in case sending and receiving sysex dumps requires a bizarre process for this
particular instrument.  I've certainly seen some hardware like that.


ALSA MIDI BUFFER

One of the problems that came up during the development of this program was
that because MIDI is very slow, it is very easy to fill ALSA's 4kB buffer
with data to transmit faster than it actually gets sent out.  Sysexy tries
very hard to avoid buffer overflow, though it will let you set transmit
speeds that will cause it if you set your transmit delay too low (so don't
do that).  Sysexy's default block size and send delay settings should be
safe and fast, so I would recommend normally not changing them.


BACKGROUND TIMERS

There are two features of Sysexy that require it to be in some sense
multithreaded, even though I decided early on that I was not going to
fork a subprocess or call external Linux executables.  Those two
features are:

1)  During transmit and receive, the user needs to be able to cancel
    or abort the transfer while it is still going on, even if Sysexy
    is hung waiting for data (especially then!), so the GUI needs to
    remain alive even while the program is busy.

2)  If an incomplete dump is received for a synth we have a support
    profile for (i.e., one that's described in Sysexy's gear database),
    we may know from the instrument description how many bytes long the
    dump is supposed to be, and thus we end up waiting for that many
    bytes to arrive not expecting that because it's incomplete, they
    will never come.  There needs to be a stall timer...which needs to
    be running while everything else also runs.

Both of these requirements mean that Sysexy can't truly be single-
threaded.  Thanks to some features provided by Tk though, the actual
Perl code doesn't need to deal with this.  Tk provides the repeat()
and after() functions, which let you setup a background handler to
either call something repeatedly each arbitrary number of milliseconds
or to do such as a one-shot operation.  Sysexy's rxsysex(), txsysex(),
and deadman() functions all heavily rely on this Tk background thread
handler facility.  Thoough Sysexy only uses two such handlers, Tk can
actually support an arbitrary number of concurrent thread timers, and
can let you control them independently from inside your program.
Cool beans, free multitasking, the Tk way...


PROGRAM ICON
The "sysexy.xpm" for Sysexy's program icon is provided as a separate file
in ~/.config/sysexy/, for any who want to manually put it in the menus
or desktop of their X11 window manager.
