I suggest that you use "egendefinert RGB". The English-Norwegian Windows glossary does not include the whole phrase, but lists the two terms separetely. There is no translation for RGB in the glossary, and the acronym is kept in Norwegian. (It might be of interest that I have found "fargeterminal" for "RGB monitor" in a dictionary, i.e. sometimes RGB is translated)

You can download the windows-glossary for free at: ftp.microsoft.com/developr/msdn/newup/glossary if you want to search for more examples.