Sysexy MIDI Librarian :
the sexy sysex librarian that Linux needs...
============================================


INTRODUCTION

One of the nifty features of MIDI is that it lets you save your data (usually
sounds and sequences) from your synthesizers, drum machines, effects processors
and other studio gear to some other device's storage.  This effectively means
you have can have more sounds, sequences, or whatever than the originating
machine's own memory could hold.  The MIDI term for this is a "sysex dump," and
in the 1980's when MIDI was new, most people didn't have computers, so it was
common to do sysex dumps from one synthesizer or sampler keyboard that didn't
have any kind of disk drive, to another keyboard in the same setup that did.
Keyboardists at that time would use anything that had a disk drive of any kind
as a sysex target for their other disk-lacking equipment's dumps, and MIDI
would let you do that -- use one keyboard to save data from other keyboards
that had no on-board way to offload theirs.

Of course, if you did have a computer, things got much better.  Computers made
immensely better filers of these dumps than any keyboard or MIDI appliance.
Software made for the purpose of doing this was known as a MIDI librarian
utility.  It handled the communication between the computer and your gear, and
loaded/saved the dumps containing all your sounds, sequences, etc. to disk.
Almost every computer platform ever, going back to MS-DOS, Commodore 64, and
other such Stone Age machines has had some kind of MIDI librarian programs.

Today almost everyone has a computer, and Linux now has a very nice selection
of studio software for sequencing, recording, sampling, sound mutilation...but
for just getting a sysex dump done, the facilities have been...underwhelming.

There is the 'amidi' command, distributed with the ALSA utils package, for
musicians who are so steeped in the culture of UNIX that they actually enjoy
running shell commands when they're trying to be in an artist frame of mind.
There is also JSynthLib, a Java program that hasn't been developed in awhile.
It wasn't specifically written for Linux, but it works anyway because it's
Java.  Do you really want to do sysex with an abandoned Java app?  Me neither.

Sysexy provides a Linux-native, compact (both in screen size and disk usage),
fully featured and user-friendly sysex librarian written specifically for
Linux users.


FEATURES

* GUI interface with keyboard shortcuts -- no more 'amidi' commands.

* Database of known gear, and MIDI commands to interact with it:
  For machines in Sysexy's database, it will remotely command the machine
  to send dumps without the need to do anything on the instrument's own
  control panel, and recognize its various types of dumps from eachother.

* Handles unknown gear as "generic", as 'amidi' command, and MIDI hardware
  appliances like Yamaha's MDF-2 or -3 would do.  In other words, it's not
  absolutely necessary that Sysexy know about your machine to get a sysex
  dump done -- unless your instrument has quirks that are impossible to
  work with generically, as many devices do.  Of course, such devices can
  be made to work if they are setup for non-generic support as a recognized
  device in the gear database (more below).  Teach Sysexy about the weird
  requirements of your instrument, and they will talk to eachother!

* Settings file stores your preferences (per synth) for physical MIDI port,
  channel, buffer size, transmit delay, verify/retries...etc.

* Gear database contains advice for dumps from hard to work with synths,
  for those instruments that are just a little eccentric.

* Recognizes your sysex dumps as coming from a particular synth when you
  scroll a highlighter bar over them, without having to even select the file.
  It knows supported MIDI machine files by "fingerprint," and will tell you
  what they are in a message window at the bottom as you scroll -- great for
  folders full of sysex files from who knows where, with unhelpful filenames.

* Verify/retry feature for devices/systems that are prone to drop or corrupt
  transfers for whatever reason.  This setting is stored per device, so that
  only your gear that needs verify gets verify, and other more reliable
  instruments you have don't waste any time on it.

* "Pseudobank" feature for synths that can't do bank dumps -- to make them
  fake it and get virtual bank dumps anyway.  (We're looking at you, DK600.)

* GUI gear database editor to let you enter the information/codes for your
  gear, and get device-tailored support for machines that Sysexy doesn't
  ship with machine profiles for yet, without having to directly edit
  config files.

* Handling of instruments with idiosyncratic/non-standard behavior --
  embedded MIDI channels inside dumps, embedded patch numbers, even
  weird CC commands, all specified in the gear database for each device.
  This makes it possible to do sysex dumps from devices that normally
  won't talk to anything that isn't specially made to receive a dump
  from that instrument model.  Needless to say, amidi doesn't do this,
  and neither do hardware appliances (at least not for dumps from
  a different manufacturer's gear).

* Written in pure Perl, executable currently taking only 117kB of disk
  space (take that, Java!).  Sysexy does not run amidi or any other
  Linux shell command under the hood.  Everything it does is Perl,
  running from a single executable.  Leave Sysexy running while you
  sequence, run your DAW, whatever.  Its GUI is small and unobtrusive,
  and it will stay out of your way.


GETTING STARTED

When you run Sysexy for the first time, it will display any ".syx" files it
finds (with the .syx extension trimmed off) in a big scrolling list that
occupies most of the program's window.  If it doesn't find any at all, it
will suggest in its message window at the bottom right that you browse to
another folder.

At the top, the current directory is displayed, with a BROWSE button to the
right that will let you do just that.  Your current directory is a saved
configuration setting, so if there is a folder where you keep all your sysex
dumps, next time you run Sysexy, you'll be there automatically, or at least
you'll be in whatever folder you were in last time.

The scrolling list has a red highlighter bar that can select files, either
by clicking on them with your mouse, or by moving up and down with your
arrow keys.  Any file the highlighter bar is on will be looked at to see if
Sysexy can tell what it is, without you having to do anything other than
move the bar up and down, and if recognized, the type of file will show in
the message window at the bottom.  This file identification is based on the
file's size and contents, and does not depend on any naming scheme.  If
you're familiar with the UNIX 'file' command, it's similar to that.

At the bottom of the main window, left of the message area, are the
action buttons, which are all pretty obvious.  DBEdit is for editing the
gear database yourself, to enter information for equipment you have that
Sysexy doesn't yet know about.  See the TECH document that accompanies
this manual for information about creating/editing gear profiles, since
doing this involves diving deeper into MIDI machine code than most
musicians really want to go.  If you don't want to get into that, don't
click DBEdit.  To quit the program, click Quit, or type Ctrl-q.


RECEIVING A SYSEX DUMP (THE PRIMITIVE CAVEMAN WAY)

Click the 'Rx' button to bring up the Receive dialog window, or type
Ctrl-r.  There is a progress bar and yet another message window to
show receive status, and many pulldown menus.  By default (and
especially if this is your first time running the program), the top
pulldown should show "Generic MIDI Device" on the Hardware Device row,
and below that "Generic manual" on the pulldown for Dump Type.  In
this default mode of operation, Sysexy is like amidi but prettier.  It
doesn't know anything about your gear, doesn't look inside incoming
dumps to try to understand their structure, doesn't presume to know
when the end of the transmission is reached (it will wait for you to
click Done or Cancel), and it can't do verify.  This is Sysexy's least
capable mode, but it will work just as you expect a typical sysex
dump utility to work, and so you can get started with it right away
without having to read much more of this manual.  Select the physical
MIDI port on your computer that your instrument is connected to in the
next pulldown, then click the big "* REQUEST / LISTEN *" button.
Sysexy will wait indefinitely while you use the control panel on your
instrument to send a dump of some kind, and when the data comes in,
Sysexy will receive it.  It will not end the transfer until you click
Done (if you want to save it), or Cancel if you'd like to throw it away.
This is all very no-frills and manual, but it works.  Congratulations,
you primitive MIDI caveman (or woman).  You did a sysex dump.


RECEIVING LIKE A CAVEMAN (BUT, A CAVEMAN WHO IS IN A HURRY)

For a slightly less primitive experience, change the Dump Type selector
to "Generic auto".  Notice that now Verify/Retries selector has become
usable.  Two things have changed here from Generic Manual mode above:

1)  You're still doing a generic dump, so Sysexy still doesn't presume to
    know how to control your device (or even know what kind of device it
    is), but it now assumes one thing:  We now assume your device sends
    sysex dumps that contain a single 0xF7 byte at the end, and we assume
    that 0xF7 byte signals the end of transmission.  (Sorry if you were
    wanting to totally not know how the sausage is made.  This is why
    Generic Manual is the default, so you don't have to know about this
    if you don't want to bother with it.)  For probably 80% or more of the
    gear you will see, this assumption is true -- most sysex dumps do
    indeed terminate with a single 0xF7 at the end, and that is the only
    0xF7 byte that will occur anywhere in the transmission.  If we can
    assume that a 0xF7 byte means we're done, we'll stop receiving
    _automatically_ when we see it, not wait for you to click DONE.

2)  Since we can make that assumption, we can go a step further, and even
    make Sysexy automatically get ready to receive again afterward, to
    verify what just came in.  This is why in Generic Auto mode, you now
    can use the Verify/Retries pulldown (more about using verify further
    on in the manual).

So, if Generic Auto is more convenient, why don't we do this all the time?
Because there are a lot of MIDI instruments out there that send dumps with
more than one 0xF7 in the same dump, which means we can't always be certain
that seeing the first 0xF7 means we're done receiving.

If all of this was more than you wanted to know, just stick to Generic
Manual mode (discussed above, the default if you didn't mess with anything),
and you should be able to handle about any dump that way -- but you'll be
missing out on a lot of Sysexy's nicer features.  For that, keep reading.


RECEIVING SYSEX LIKE A MODERN HUMAN

Of course, if Sysexy actually does know about your instrument in its database
of supported devices, you can select the actual brand and model of instrument
in the Hardware Device pulldown, and then we can really see what this program
can do.  (Generic is so...generic.)

Select the type of dump you want under Dump Type.  The most typical choice
here is "Bank", for a dump of all your patches/sounds from the whole memory
of your instrument, sent to you as one big file, which you can later send
back to the instrument to overwrite everything currently in its memory with
the bank you sent.  If your instrument has some unusual way of sending
bank dumps, Sysexy now knows about it from the gear database profile for
this instrument, and it will handle the mechanics of this for you.  It will
even remotely command your instrument to send without you having to even do
anything on its own control panel, using the manufacturer's provided MIDI
dump request command for that instrument.  (We're not in generic mode anymore.)

You may also be able to select other kinds of dumps, depending on what's
possible on your instrument:  "Patch", for single sounds (see section on
PATCH DUMPS); "Config", for machines that can send their settings in a kind
of dump that's separate from the sounds themselves; "All", for machines that
can roll up absolutely everything and send it all together; and there are
some "Pseudobank" dump types that are an invention of Sysexy itself,
discussed more later.  There can be all kinds of sysex dumps, and it depends
really on what your instrument is able to send.  You will see your options
when you look at the Dump Type pulldown, so choose one now.

There is an "Advice" button to the right of the dump type pulldown, which
will usually be grayed out and inactive.  If this button is actually blue
and clickable for your instrument, that means there are advisories in the
gear database about doing sysex dumps on your device.  You should click
the Advice button to bring up a window that describes any arcane knowledge
that may be helpful, since some instruments are just a little weird, and
even Sysexy may not be able to help you if you don't know the
idiosyncrasies of your bizarre synthesizer.

As with Generic mode, select the computer MIDI port to use.  There is a
small pulldown to the right of the port selector pulldown that _may_
let you also choose the MIDI channel your instrument is on.  Probably
this pulldown will be grayed out, but some instruments need it, and if
setting the MIDI channel is necessary for yours, then the pulldown will
be usable.  (See the TECH document if you want the gory details.)

Click * REQUEST / LISTEN * now to actually receive your dump.  Unlike in
generic mode, this time, we don't need to do anything on your instrument's
own control panel.  Sysexy should send a MIDI dump request command, and
your instrument should automatically respond with the requested type of
sysex dump.  The status and progress bar displays should show the data
coming in, and Sysexy will know when the dump is complete.  If there was
any kind of transmission error, it will let you know.  (See the VERIFY
section if you want Sysexy to actually do something about such errors
besides warn you.)

After it's done, as with generic dumps, click Done if you want to save
the file, or Cancel to throw it away.  Whenever you successfully save a
sysex dump, Sysexy will save the settings you used with that instrument
in a per device configuration in Sysexy's settings file, so you don't
have to keep fiddling with these pulldowns every time you receive a
dump from a synth that you've already received from before.  Next time
you select that synth in the Hardware Device pulldown, all your previous
transfer settings will be recalled as default options automatically.


THE UNCERTAINTY AND DOUBT OF RECEIVING SYSEX DUMPS
LIKE A MODERN HUMAN IN OUR COMPLEX, MODERN WORLD:
(IN OTHER WORDS...YOU WANT VERIFY!)

But what if you have doubts?  More specifically, what if you have some
piece of gear that drops bytes when you receive from it, risking the
corruption of your sounds?  Generally speaking, if this happens a lot,
you need to investigate your cabling, since generally this isn't really
supposed to happen, and it's probably a dumb idea to try to work around
it with software when you probably have a hardware problem.  However, when
it comes to MIDI equipment (particularly vintage MIDI equipment), there is
no single rule that is right all the time.  I personally have a vintage
synthesizer with a processor that can't quite keep up with its own sysex
dumps, and it loses bytes about 30% of the time that it tries to do a bank
dump.  It is not my cabling, not my computer, and not a problem with my
settings.  This particular instrument just can't quite keep up with its
own data stream, because it was made around the dawn of time, by primitive
people using stone knives and bear skins, and that's just the way it is on
that particular synth.  (But it does sound amazing, so I ain't losin' that
board, ya know -- the things we do for music.)

For these situations, where you receive your sysex dump but you have
doubts, Sysexy has a verify/retry feature.  This setting is saved per
instrument, so it will only be active for gear that you need to use it
with if you selected it last time.  In the Verify/Retry pulldown,
change the default setting of "Never" to either "Receive", if you'd
like to only verify data being saved on the computer but you don't
care about verifying transmits to the synth, or "Always" if you'd
prefer to always verify in both directions.  On the one hand, my own
problem instrument only has trouble sending to me, but never seems to
corrupt when data is sent to it, so the "Receive" setting is useful in
my own case.  However, be wary of possible situations that may arise in
your own use where you may corrupt while sending, receive the now-munged
data back to the computer later (which will pass "Receive" verify ok,
because the corruption happened earlier), and save to your hard drive
without ever knowing it became corrupted.  If you're not certain that
your device only corrupts in one direction, it's probably better to just
select "Always" here because of this.

The little pulldown to the right lets you choose the number of retries
before giving up, from 1 to 10, or you can write in a number yourself.
Sysexy considers receiving the same data twice in a row to be a verify,
and will declare success as soon as that happens.  You can of course
abort with the Cancel button at any time.


SINGLE PATCH DUMPS

They say the Devil is in the details, and that's very true for the
seemingly simple task of sending a single sound back and forth between
a synth and a computer.  Sending whole banks of sounds is actually much
simpler.  Sysexy can do single patch dumps (usually), but here's where
you may have to know a little about the mechanics.

Some synthesizers have what's called an "edit buffer," which is a place
where the sound lives while you're fiddling with it.  It may have started
out as a saved preset in its memory, but you started adjusting the sound
and trying to make a new patch that sounds completely different, so it's
no longer the saved sound you started with.  That original sound is still
saved because you haven't overwritten it, so where is this new thing that
you're creating located?  The edit buffer is the limbo zone where sounds
still being made live while you're still making them.  What does all this
have to do with single patch dumps?

Some synthesizers will stick an incoming single patch dump into their edit
buffer when they come in.  Then it's up to you to decide where you want to
save it in that instrument's memory, usually using about the same menu
commands that instrument would make you use if you were saving a sound you
just finished making by manual fiddling on the control panel.  If every
instrument handled incoming single patch dumps this way, Sysexy wouldn't
have to do anything special, and the librarian's job would be so easy.

But that's the simple case, and not all synths do that.  Some synths
actually write the incoming patch directly into their memory immediately
upon reception, as an actual saved preset.  But where?  How do they know
where it goes if you don't save it yourself on the instrument's control panel?

They expect the destination location to be embedded in the dump, at a byte
offset that is particular to that make and model of instrument, using an
encoding scheme that is similarly particular to that instrument.  (I warned
you that this was going to get ugly.)  How would a generic sysex utility
know how to send this?  One wouldn't, and they can't.

Naturally, Sysexy can do single patch dumps in the first, simple case,
where the choosing of a destination preset location is done by you on the
synth's own control panel.  Also, for supported devices that are in the
gear database, we _will_ handle single patch dumps _if_:

- the destination memory location is stored in a single byte.  The gear
  database will tell Sysexy which byte in the dump contains the destination.

Since MIDI data is 7-bit, that means the destination can have a range of
0 to 127.  But what about synths with banks bigger than that?  What about
synths that have more than one bank?

Sorry, dude.  Perhaps a future version of Sysexy will attempt to deal with
this madness, but currently, Sysexy only does single patch dumps from synths
that either do not need embedded destination bytes, or ones which require a
single destination location byte.  We do not handle cases where there may be
more than 128 possible destinations, written into more than one embedded byte
inside the dump, mostly because of how many ways that could be possibly be
done by an instrument manufacturer, and the difficulty of having one program
be able to adapt to them all.

When the Dump Type dropdown selects Patch, if this synth needs a destination
byte, you will be able to choose the patch number to request in the bottom
field, labelled "Patch # / PgmChg send".  The PgmChg button to the right of
it will be active, and if you click that button, it will send a patch change
command to your instrument immediately, to call up the patch corresponding to
that particular byte number.  The reason you might want to do that is to deal
with the common dilemma you'd have in this situation, where you want to sysex
a particular patch, but you have no idea which patch in that synth's memory
correlates to which byte value.  You can click the button, and a MIDI program
change command will select the patch on the synth that has that number, so
you won't be cruelly surprised when you find out later that the number was
referring to a whole different sound than you thought it was.

When you have the correct patch number selected (if your synth even needs
that), click * REQUEST / LISTEN *, and your instrument should send the patch.


THEY SAY IT IS BETTER TO TRANSMIT THAN TO RECEIVE

Select a sysex file with the red highlighter bar in the main window, and
click 'Tx', or hit Enter or Ctrl-t.  Again, here as in the Receive window,
if your synth is weird enough that the gear database has an advisory about
it, the Advice button will be active in the upper right corner.  The
Transmit dialog does not have a pulldown to choose an instrument type like
the Receive dialog does, because it will identify the file you highlighted
and know the appropriate instrument from that, using whatever settings you
may have used during previous dumps from/to that instrument.  If the file
is of an unknown type, then we're essentially in generic mode now, just as
if you had chosen Generic MIDI Device in the Receive window's pulldown.

Choose the port (and if necessary, the MIDI channel).  Now, here in the
Transmit window we have two new fields, "Buffer (bytes)" and "Delay (ms)".
These two settings can probably be left alone completely, since they are
at defaults that should be safe and efficient, but if you really want to
experiment, go right ahead, keeping in mind that corrupting data is quite
possible with the wrong speed and block size settings.

Pseudobank is discussed in its own section in the TECH manual, and it
should not be necessary to use in 99% of cases anyway.

Verify/retries works about the same here as in Receive, with the exception
that rather than two identical consecutive transfers meaning success as it
does with Receive, a Transmit verify works like this:

Sysexy will send the data once, and only once.  It will then ask your
instrument to send it back, and it will ask twice to make sure it came back
correctly.  Basically, a send, and then two receives, or however many
receives it takes to get two consecutive identical receives.  If you hit
the retry limit, the process starts over again, with a fresh transmit,
followed by at least two receive attempts, etc., up to your retry limit.
Basically, if you're using verify, Sysexy will try very hard to make sure
you do not corrupt your sounds going in either direction.

The Patch # field and PgmChg buttons work as with their Receive window
counterparts, but of course, are only active if you've highlighted a file
that is a patch dump (special exception for pseudobanks, described elsewhere).

As with the Receive window, all these settings get saved if you actually do
a transmit, and will be used again the next time you transmit or receive a
dump to/from the same synth again, as defaults for that instrument.

Click Xmit when you're ready, and a progress bar window will appear to let
you see the transfer happening.


PSEUDOBANKS

[See the TECH manual for information on this subject.]


CONCLUSION

Enjoy your fun new sexy life of doing sysex dumps all day long!  But
don't have _too_ much fun.  You have to go to work in the morning.

