I'm developing a piece of software for easily modifying instruments in a MIDI file. For the most part, I scan through the track data bytes looking for program changes (C0, C1, etc up to CF). For percussion, I currently instead look for 99 for the "note on" event on channel 10. However, during testing, I have come across several files where they only use a single "99" event near the beginning of the track, and the rest of the notes are... magic? This is where I need help. Here is a sample piece of one of the files. Various MIDI editors can show me all the note on/note off events, but I can't figure out how to go through the hex and decipher it myself. I've pored through a bunch of different MIDI specification sites, and they all mention everything always having note on and off, or sometimes using note on and then velocity = 0 to turn off a note on the percussion channel. But here, there is only the single note on, yet the percussion plays as expected.
I know I have a bass drum (24) and I know I have a closed hi-hat (2a). However, there is only a single note-on (99) for the bass drum, and all subsequent notes/drums just "happen", and there are no specified note off events.
It seems it is probably a case of this description: "Running status. MIDI files actively use concept of running status. This technique allows don't store status bytes of consecutive events of the same type. It means that status byte 144 can be written only once for the first Note On event and you will not find it further in the file."
The actual file in question is 1999.midi from http://samples.mplayerhq.hu/karaoke/
Can someone break out the first few notes/drums and explain what each of the bytes is doing, and why it works without the 99 for each one? Thanks.
99 24 64 01
2a 63 1d 24 20 01 2a 20
5a 2a 43 1e 2a 20 5a 2a
5f 1e 2a 20 5a 2a 5b 1e
2a 20 58 2a 57 01 24 64
1d 2a 20 01 24 20 59 2a
4f 1e 2a 20 5b 2a 63 1e
2a 20 5b 2a 63 1e 2a 20
58 2a 63 01 24 64 1d 2a
20 01 24 20 5a 2a 63 1e
2a 20 5b 2a 63 1e 2a 20
59 2a 5f 1e 2a 20 59 2a
63 01 24 64 1d 2a 20 01
24 20 59 2a 63 1e 2a 20
50 26 52 0c 2a 63 12 26
20 0c 2a 20 59 2a 63 1e
2a 20 59 2a 63 20 31 64
I know I have a bass drum (24) and I know I have a closed hi-hat (2a). However, there is only a single note-on (99) for the bass drum, and all subsequent notes/drums just "happen", and there are no specified note off events.
It seems it is probably a case of this description: "Running status. MIDI files actively use concept of running status. This technique allows don't store status bytes of consecutive events of the same type. It means that status byte 144 can be written only once for the first Note On event and you will not find it further in the file."
The actual file in question is 1999.midi from http://samples.mplayerhq.hu/karaoke/
Can someone break out the first few notes/drums and explain what each of the bytes is doing, and why it works without the 99 for each one? Thanks.
You are correct that it is due to Running Status. If you are at a point in the MIDI file where you are expecting a status byte (hex 80 to FF), but you get a data byte instead (hex 00 to 7F), then the previous status byte applies. However, be aware that only status bytes that contain a channel (status bytes hex 80 to EF) are eligible for Running Status. The other three possible "status" bytes in a MIDI file (hex F0, F7, or FF) must always appear, and then the next status byte after that has to appear because Running Status was ended.
As you mentioned, there are two ways to specify the end of a note. A Note On with Velocity 0 indicates the end of a note. A Note Off also indicates the end of a note and lets you specify a "Note Off velocity". Either way is valid.
When you use both Running Status and Note On with Velocity 0 to indicate the end of a note, then you can use one status byte to start a run of notes for a channel.
Let me warn you against simply scanning a MIDI file for a particular hex value. That might happen to work sometimes, but it could catch unwanted matches! For example, hex 99 might appear in a Tempo meta event, or might appear as part of a size value in a track header, or might appear as a delta time value. And as you found out, when Running Status is used, you can't find the events that used Running Status with just a simple scan for the status byte. In order to work reliably, you will have to parse every byte of the MIDI file to tell for certain where every event actually starts.
In the following example, I explained the bytes for the beginning of the DRUMS track in the file 1999.midi. I indicated the use of Running Status with ditto marks (" " " " ). Remember, for reliability, you always have to start parsing from the beginning of the MIDI file, so think of this example as showing a parsing that was already in progress, continuing at the beginning of DRUMS track header.
Note that Running Status is usually used to save space, but using it is optional. In this example, some of the Control Changes could have used Running Status to skip the status byte, but they didn't for some reason. This is valid.
Be sure to read up about delta times and variable length quantities. In this example, the events are close together and the delta times all happen to be one byte. But delta times use a special encoding pattern the spec calls a "variable length quantity" that can be one to four bytes long.
If you have read about MIDI files on other websites, be sure to also check out the official specification on midi.org for comparison (login required to download):
https://www.midi.org/specifications/file-format-specifications/standard-midi-files
Other websites can describe the specs in a more friendly way, but they might get some details wrong, so it's good to compare with the official specs.
As you mentioned, there are two ways to specify the end of a note. A Note On with Velocity 0 indicates the end of a note. A Note Off also indicates the end of a note and lets you specify a "Note Off velocity". Either way is valid.
When you use both Running Status and Note On with Velocity 0 to indicate the end of a note, then you can use one status byte to start a run of notes for a channel.
Let me warn you against simply scanning a MIDI file for a particular hex value. That might happen to work sometimes, but it could catch unwanted matches! For example, hex 99 might appear in a Tempo meta event, or might appear as part of a size value in a track header, or might appear as a delta time value. And as you found out, when Running Status is used, you can't find the events that used Running Status with just a simple scan for the status byte. In order to work reliably, you will have to parse every byte of the MIDI file to tell for certain where every event actually starts.
In the following example, I explained the bytes for the beginning of the DRUMS track in the file 1999.midi. I indicated the use of Running Status with ditto marks (" " " " ). Remember, for reliability, you always have to start parsing from the beginning of the MIDI file, so think of this example as showing a parsing that was already in progress, continuing at the beginning of DRUMS track header.
4D 54 72 6B (start of track header: "MTrk")
00 00 27 7C (track size: 10108 bytes)
00, FF 21 01 00 (+0 ticks, MIDI Port 1)
00, FF 03 05 44 52 55 4D 53 (+0 ticks, Track Name "DRUMS")
00, C9, 00 ( +0 ticks, Program Change Channel 10, Program 1)
00, B9, 07 78 ( +0 ticks, Control Change Channel 10, Controller 7 Volume Value 120)
00, B9, 0A 40 ( +0 ticks, Control Change Channel 10, Controller 10 Pan Value 64)
00, B9, 79 00 ( +0 ticks, Control Change Channel 10, Controller 121 Reset All Controllers Value 0)
00, 5D 00 ( +0 ticks, " " " " Controller 93 Chorus Value 0)
00, 5B 28 ( +0 ticks, " " " " Controller 91 Reverb Value 40)
00, 99, 24 64 ( +0 ticks, Note On Channel 10, Key 36 Velocity 100)
01, 2A 63 ( +1 tick , " " " " Key 42 Velocity 99)
1D, 24 00 (+29 ticks, " " " " Key 36 Velocity 0)
01, 2A 00 ( +1 tick , " " " " Key 42 Velocity 0)
5A, 2A 43 (+90 ticks, " " " " Key 42 Velocity 67)
1E, 2A 00 (+30 ticks, " " " " Key 42 Velocity 0)
5A, 2A 5F (+90 ticks, " " " " Key 42 Velocity 95)
...
Note that Running Status is usually used to save space, but using it is optional. In this example, some of the Control Changes could have used Running Status to skip the status byte, but they didn't for some reason. This is valid.
Be sure to read up about delta times and variable length quantities. In this example, the events are close together and the delta times all happen to be one byte. But delta times use a special encoding pattern the spec calls a "variable length quantity" that can be one to four bytes long.
If you have read about MIDI files on other websites, be sure to also check out the official specification on midi.org for comparison (login required to download):
https://www.midi.org/specifications/file-format-specifications/standard-midi-files
Other websites can describe the specs in a more friendly way, but they might get some details wrong, so it's good to compare with the official specs.
Thank you. That is precisely the breakdown I needed to help me understand what was happening. Until yesterday, I was parsing the whole file, but ignoring most of the data besides locating the headers and using them to determine track start and then going until I reached track end, and then starting again when I find the next track. As you mentioned, I end up with mostly working code, and many files I tested worked great, but some files/things get erroneous data. I implemented scanning for (and mostly excluding) the data from META events today (solved finding instruments in the lyrics, text markers, etc) but figured there may be a few I actually need to pay attention to. So now I at least symbolically parse the entire header and META events, so I don't accidentally catch any non-note data.
This will be super helpful. The other key info I was missing is delta time. I couldn't wrap my head around what the values were for in the hex code without seeing it laid out like this. Makes much more sense now, and I'll read up on delta time, and implement reading all of the actual note info into my program.
expecting a status byte (hex 80 to FF), but you get a data byte instead (hex 00 to 7F), then the previous status byte applies.
This will be super helpful. The other key info I was missing is delta time. I couldn't wrap my head around what the values were for in the hex code without seeing it laid out like this. Makes much more sense now, and I'll read up on delta time, and implement reading all of the actual note info into my program.
So I've made very good progress. I can now nearly fully process each file, with all note-on/off events, and process all other events (keeping data relevant to what I am doing when needed) but one file in particular is giving me trouble, and it seems to be with a SYSEX event (or a false SYSEX event if I missed something prior to it). Additionally, there seems to be a bug in the debugger (go figure) so when I load this particular midi, I can't run the debugger to trace where it goes wrong. So now I am trying to visually parse it to see what the issue might be. There is a SYSEX event that occurs before this one. It starts on F0 and ends on F7, as expected. However, in the code below, there is an F0 with no F7 for the rest of the file, which I believe is what causes a buffer overrun when trying to locate the end of the SYSEX event. I'm having trouble visually processing it to see if my program is missing something (which would be great if I could actually debug it)
Can you properly break this down for me after the B3 5D 00 controller change (if that is indeed correct?) This will hopefully allow me to correct any programming mistakes in my program that might be reading too few or too many bytes and throwing the rest of the file out of whack.
83 24 0B 7F 00 FF 2F 00 (end of previous track)
4D 54 72 6B 00 00 0D 47 (MTrk + track length)
00 delta time 0
FF 03 06 42 52 41 53 53 31 (Track name 6 bytes BRASS1)
2D
B3 5D 00 (ctrl change @ delta 2D?)
00
0B 7F << (is this a running mode ctrl change also?)
00
5B 50 (then I get lost)
5D 07 6E 00
0A 40 D1 36 00 00 00 20
02 20 C3 3C 00 B3 07 73
9F 53 93 42 64 0F 46 6E
0E 47 7F 01 42 00 0F 46
00 83 50 47 20 00 45 64
5A 45 00 00 42 64 5A 42
00 00 3E 64 3C 3E 00 F0 << (then mysterious F0 in question that I believe triggers my sysex buffer error)
40 42 7F 83 60 42 00 00
40 7F 00 B3 0B 7F 0A 0B
7E 06 0B 7D 07 0B 7C 0A
0B 7B 07 0B 7A 06 0B 79 Can you properly break this down for me after the B3 5D 00 controller change (if that is indeed correct?) This will hopefully allow me to correct any programming mistakes in my program that might be reading too few or too many bytes and throwing the rest of the file out of whack.
In a MIDI file, the F0 event that stores a System Exclusive event is immediately followed by a variable length quantity that indicates how many data bytes follow. In some cases*, the last byte might not be F7. Because of this, you have to use the length amount to tell where the message ends, you can't scan for F7.
(* The MIDI File specification lets you store a System Exclusive message in multiple packets. The first packet is in an F0 event to start the message, and the other packets are in an F7 event to continue the message after a specific delta time delay. Only the last packet is required to end with F7 as its final byte.)
May I suggest you open the MIDI file in a MIDI sequencer and open an Event List window? I like to use Sekaiju because its Event List view shows every event as it exists in the file. Then you can compare what you see in the Event List to the bytes in the MIDI file. (Be aware that other MIDI sequencers will often combine, move, or hide events to be more friendly and less technical, but this makes it harder to compare what you see in the Event List to the bytes of a MIDI file. I like Sekaiju because it shows all the events as they exist in the MIDI file.)
As you know, the actual bytes of a Format 1 MIDI file are grouped by tracks, and within each track the events appear in time order for that track. But in Sekaiju (and other MIDI sequencers), if you view more than one track in the same Event List window, Sekaiju shows the events for the tracks mixed together and sorted by event time. So you will want to be sure to view just one track at a time in an Event List window, then you will see just that track's events in the same order as the MIDI file bytes for that track.
(* The MIDI File specification lets you store a System Exclusive message in multiple packets. The first packet is in an F0 event to start the message, and the other packets are in an F7 event to continue the message after a specific delta time delay. Only the last packet is required to end with F7 as its final byte.)
May I suggest you open the MIDI file in a MIDI sequencer and open an Event List window? I like to use Sekaiju because its Event List view shows every event as it exists in the file. Then you can compare what you see in the Event List to the bytes in the MIDI file. (Be aware that other MIDI sequencers will often combine, move, or hide events to be more friendly and less technical, but this makes it harder to compare what you see in the Event List to the bytes of a MIDI file. I like Sekaiju because it shows all the events as they exist in the MIDI file.)
As you know, the actual bytes of a Format 1 MIDI file are grouped by tracks, and within each track the events appear in time order for that track. But in Sekaiju (and other MIDI sequencers), if you view more than one track in the same Event List window, Sekaiju shows the events for the tracks mixed together and sorted by event time. So you will want to be sure to view just one track at a time in an Event List window, then you will see just that track's events in the same order as the MIDI file bytes for that track.
Here are some additional points:
1. Have you learned how to tell if a variable length quantity is more than one byte?
If you are expecting a variable length quantity and you see a byte from hex 80 to FF, then the variable length quantity continues for more than one byte. Keep processing bytes until you see a byte from from hex 00 to 7F, which is the last byte of the variable length quantity. (A variable length quantity can be up to four bytes.)
To turn a multi-byte variable length quantity into the value it represents, you'll have to use bitwise manipulation to concatenate the least significant 7 bits of each byte. (The first byte contains the most significant bits of the value.)
2. In your first post, I noticed that whatever process you are using to look at the hex bytes seems to change hex 00 to hex 20. In your most recent example, I think that might still be happening at least one time.
1. Have you learned how to tell if a variable length quantity is more than one byte?
If you are expecting a variable length quantity and you see a byte from hex 80 to FF, then the variable length quantity continues for more than one byte. Keep processing bytes until you see a byte from from hex 00 to 7F, which is the last byte of the variable length quantity. (A variable length quantity can be up to four bytes.)
To turn a multi-byte variable length quantity into the value it represents, you'll have to use bitwise manipulation to concatenate the least significant 7 bits of each byte. (The first byte contains the most significant bits of the value.)
2. In your first post, I noticed that whatever process you are using to look at the hex bytes seems to change hex 00 to hex 20. In your most recent example, I think that might still be happening at least one time.
2. In your first post, I noticed that whatever process you are using to look at the hex bytes seems to change hex 00 to hex 20. In your most recent example, I think that might still be happening at least one time.
I noticed that too. I am copying the hex values, presumably as text, straight from my hex editor, but for some reason when I paste as regular text, it turns 00 in to 20 (everything is correct in the hex editor though). I tried to catch them all this time, but guess I missed one
I am jumping between multiple editors, and that one is the only one that will let me highlight and copy the hex as regular text. All the others will let me edit, but not select/copy.I have hopefully gotten the variable length values figured out. I do process them, but I have yet to check if they are producing the correct values (but they seem to be at least reading the correct number of bytes.) For the most part, I do not need this information, I just need to make sure the read buffer ends up in the correct location.
I did not know the sysex events specified length. I'll add that check in.
I also have a bunch of different midi editors that offer event views, but as you mentioned a lot of them make it too viewer friendly. I have Sekaiju already, but hadn't checked the event viewer in it, as I had tried to use it for something else previously (changing instruments!) and it couldn't do it how I wanted. I'll definitely have a look at the event viewer in there! I also just found a program called Reaper that seems to also have a good event viewer.
Is there a pre-made quick chart somewhere that lists all of the midi commands and number of bytes for each in a handy cheat-sheet style? The official ones are super detailed and have too much info for a quick reference. I haven't yet found one that is laid out in the most basic form where I can quick glance to see I'm reading the correct number of bytes (though by this point I think I do have them all figured out).
Is there a pre-made quick chart somewhere that lists all of the midi commands and number of bytes for each in a handy cheat-sheet style?
In a MIDI file
hex 80 to BF: 2 data bytes
hex C0 to DF: 1 data byte
hex E0 to EF: 2 data bytes
hex F0, F7, FF are variable length:
F0 length data
F7 length data
FF type length data
where
type is 1 byte
length is a variable length quantity (1 to 4 bytes)
data is the number of bytes indicated by the length value.
REFERENCES
The status bytes with a channel -- hex 80 to EF -- work the same way as the status bytes on a MIDI cable with the same hex value. And only these status bytes with a channel are eligible for Running Status. MIDI 1.0 Detailed Specification, PDF page 73 (printed page T-1) "Table I: Summary of Status Bytes" shows the number of data bytes for MIDI cable status bytes. (For the status bytes usable in MIDI files, only look at the Channel Voice Messages in that table.)
The other three "status" bytes possible in a MIDI file -- hex F0, F7, or FF -- have variable length data. (I call them "status" bytes because they don't work the same way as the status bytes on a MIDI cable with the same hex value.) Standard MIDI Files 1.0, PDF pages 8 and 9 (printed pages 6 and 7) explains the general format of the F0, F7, and FF events.
I did not know the sysex events specified length.
Be aware that System Exclusive messages work a little differently on a MIDI cable and in a MIDI file.
On a MIDI cable, you would scan for System Exclusive events by scanning for F0 as the start and scanning for F7 as the end. (Or if you encounter any other status byte with a channel -- 80 to EF -- that indicates an abnormal end to the System Exclusive message.)
In a MIDI file, you need to know the number of data bytes in advance without scanning, so a variable length quantity is used just after the F0 status byte to indicate how many message bytes follow.
I've manually parsed the file on paper up to the offending F0, which landed right where it would then be read as a SYSEX. This means my program is indeed reading all of the correct number of bytes up to this point. Then it reads the F0, and looks for the end of the SYSEX event.
None of the event viewers (including Sekaiju) show a SYSEX message at this point.
00 | "" 3E 64 Note on
36 | "" 62 00 Note off
F0 40 !!!!
So I think I am prematurely capturing the F0 as SYSEX, even though in this case it appears to be delta time for the next note instead. I process status FF, F0, and F7 before anything else, so it looks like I may need to tweak that code. My brain hurts
***** Correction: My assumption was incorrect. I went in to my program, modified the part that makes it crash so it doesn't crash, and added a bunch of code to output my processed data into a debug text file. There seems to be a problem with how I am handling running mode. Every time it kicks in, it seems one too few bytes are being read (at least in this particular problematic track). So when I compare to my manual parsing, the bytes slowly drift off, and everything becomes part of delta time for a while until running mode ends. So I have to track that down and see where it's going wrong.
None of the event viewers (including Sekaiju) show a SYSEX message at this point.
00 | "" 3E 64 Note on
36 | "" 62 00 Note off
F0 40 !!!!
So I think I am prematurely capturing the F0 as SYSEX, even though in this case it appears to be delta time for the next note instead. I process status FF, F0, and F7 before anything else, so it looks like I may need to tweak that code. My brain hurts
***** Correction: My assumption was incorrect. I went in to my program, modified the part that makes it crash so it doesn't crash, and added a bunch of code to output my processed data into a debug text file. There seems to be a problem with how I am handling running mode. Every time it kicks in, it seems one too few bytes are being read (at least in this particular problematic track). So when I compare to my manual parsing, the bytes slowly drift off, and everything becomes part of delta time for a while until running mode ends. So I have to track that down and see where it's going wrong.
Thanks in a major part to this thread, my first version of my software is complete. I've tested it quite a bit, and sent it off to a friend to try out. It does everything I set out to do, so from here it will be bug fixes (if any) and feature additions.
I'm able to load any standard MIDI file (that I've tried ) and easily change any instrument to any other instrument, either per channel or per song. They can be changed based on their original assignment when loaded, or their current assignment (ie, you change a piano to an organ, and there is already another organ: you can then change them again independently, or change all of that organ at the same time). Percussion can also be changed the same way.
Features that I may add in the future:
- support for bank selection (currently any bank selection remains untouched, so whatever was their originally will carry over to the new file)
- support for saving / loading and applying batch instrument changes (good for use with video game music converters where many songs will have the same instrumentation)
- support for changing percussion to a standard instrument and vice versa
I'm able to load any standard MIDI file (that I've tried
) and easily change any instrument to any other instrument, either per channel or per song. They can be changed based on their original assignment when loaded, or their current assignment (ie, you change a piano to an organ, and there is already another organ: you can then change them again independently, or change all of that organ at the same time). Percussion can also be changed the same way.Features that I may add in the future:
- support for bank selection (currently any bank selection remains untouched, so whatever was their originally will carry over to the new file)
- support for saving / loading and applying batch instrument changes (good for use with video game music converters where many songs will have the same instrumentation)
- support for changing percussion to a standard instrument and vice versa
- Page :
- 1
There are no replies made for this post yet.
