I've played a lot of niche games thanks to me getting into RPGs early on in my life back in the 90s. So here's a few some people may or may not have heard of:
Radiata Stories - PlayStation 2
A game made by tri-Ace, it uses a similar combat system to that of Star Ocean: Till The End of Time, but with a new limit break bar. Instead of controlling characters manually, you give them orders, and formations to follow. The game is for the PlayStation 2, and offers a refreshing take on tri-ace's action-based RPGs. The game doesn't have too much of a complicated story, and has two sides of the same story, so you have replay value.
Tales of Destiny - PlayStation
This was my introduction to the wide world of niche RPGs. The game was the first Tales game to come to North America, with Tales of Phantasia on the GBA to follow suit after. It had an in-depth story, and a fake-out plot point making you think it was the end, but it wasn't, which has been a staple in main series tales games for a long time now. The game inspired the story of Tales of the Abyss in the sense of "What if the bad guys won?" A director's cut was made for Japan Only on the PlayStation 2.
Legend of Legaia - PlayStation
Another niche RPG, but this time it combined the fighting mechanics of an arcade fighter, with the wonderful world of JRPGs. The story is a bit cliche in some respects, but it's a good early RPG. There is a "Legaia 2," but like Final Fantasy, it has nothing to do with this first game.
Xenogears - PlayStation
An extremely mature JRPG that uses some unique fighting mechanics similar to an action game. You input commands to deplete a gauge or point system. IE: One action may cost 2 AP, while another costs 3 AP. AP regains each turn and you can guard/etc. without losing any. It's a bit difficult for some people to play this game, but it has its grinding areas well defined. This was the first Xeno- game. Xenosaga was the next serial to spawn, then Xenoblade Chronicles. The story within this game is quite fragmented in certain parts, and some of it is locked behind optional content. Square still owns Xenogears.
Valkyrie Profile - PlayStation/PSP
Valkyrie Profile was a good game. Squaresoft however didn't know how much people would love it, and thus there was limited production of the game. So little in fact that it became one of the rarest and most expensive JRPGs. Its PSP re-release/remaster would follow the same, and it became rather rare. The game has you play as Lenneth, a valkyrie who is out to gather Einherjars to fight in Ragnarok. There are multiple endings (Each ending has a letter assigned to it.) Combat follows a similar style to Xenogears, but this time, characters are assigned to buttons instead of just attacks. IE: Triangle could be one guy, Square could be Lenneth, etc.
Dragon Quest Series - Famicom/NES, Super Famicom/SNES, PlayStation, PlayStation 2, DS, 3DS, etc.
This series is beloved by many people for its simplistic stories and call backs to ancient JRPG mechanics. The Dragon Quest series to this day still uses tried and true fantasy storytelling. A dark evil must be destroyed, a damsel saved, a world restored. This series has not been touched by the complex nature of other serials, such as Final Fantasy. it is a great game series to play to get away from complexity. The popular classic JRPG only recently started to get ground again in the west after Dragon Quest VIII.
Golden Sun - GBA
A JRPG that took on the theory of flat earth, Pangea, and intertwined it with magic. A solid RPG developed by Camelot, which has a great music track composed by Motoi Sakuraba, who also does music for the Tales series, and other Tri-Ace titles. It uses a summon junction system similar to Final Fantasy VIII to raise stats and cast elemental spells.
Grandia Series - PlayStation, PlayStation 2
The Grandia series is one of my earliest serials in the RPGs I've played. The first Grandia was great, and told the story of a young boy wanting to go on a grand adventure, he finds love in the process, and a dark secret of the world. Grandia II is also really good, and its recently released Steam version is a great Dreamcast port. If only Grandia III could come out on the PSN or PC/Steam.
Wild ARMs Series - PlayStation, PlayStation 2, PSP
The Wild ARMs series is a JRPG series that takes place in a wasteland of sorts. Not all is as it seems in most cases however. Usually the story tells of a time when everything was right, the land was bountiful with flora and fauna. Wild ARMs 2, Wild ARMs: Alter Code F (Remake of the first), and Wild ARMs 5 happen to be my favorites of this series. Wild ARMs XF (A tactical RPG) was the last entry, on the original PSP.
Breath of Fire Series - Super Famicom/SNES, GBA, PlayStation, PlayStation 2, Mobile
The Breath of Fire series was Capcom's attempt to make a good JRPG series. It was a great JRPG series. The game usually centered around a boy who has the power of a dragon, or is connected to dragons in some way. My favorite game of this series is Breath of Fire III. The last main series game was released in Japan on Mobile Only, due to the fifth title being such a shock and disappointment.
Mega Man Legends and The Misadventures of Tron Bonne - PlayStation
The Mega Man Legends series was a short-lived PlayStation series made by Capcom. It is, in a sense, the first good RPG Shooter. The game's series was meant to have a conclusive "Mega Man Legends 3," but development halted / and was thrown out. The games are still loved by many people, me included.
Thousand Arms - PlayStation
The first ever Dating Simulator RPG to come to the western market on a major console. Made by ATLUS, it had a very raunchy story. it wouldn't be until...
Ar Tonelico Series - PlayStation 2, PlayStation 3
...the Ar Tonelico series that the western major console market would get to see a new Dating Simulation RPG. It's a pretty good series, but the voice acting can leave you wanting.