Is it to late to get into racing?

No

Expect to spent a lot of time at the track before you are ready to race. Hold your line on the jumps...

Of course not. Look where you want to go. Watch the lines fast guys take. Watch lots of racing. Session sections when you can safely. Take a section and hit it over and over again at various levels of intensity get technique down first then speed. 

It’s never too late. Go to the track and find someone who is racing in your beginner class and ask them how it works. They will more than likely tell you everything you need to know. 
Also, there’s a guy named Brandon that you should probably look up on YouTube.  It was his first race. 

All I can say, is don't be like Brandon. That video was unfortunately hilarious even though he crashed a lot.

I'll lust leave this here in response to OPs question.

I started racing at 23. I couldn’t clear a table top and would finish last place in the beginning but slowly worked at it and got better to the point of reaching A grade. The number one thing is not to get disheartened when your getting beaten badly in the beginning and take the small victories of progression. Secondly the biggest issue I saw with other guys similar pace but not progressing is intensity when you ride, every time your at the practice track ride with intensity and purpose. 

Anything you want to do and are physically and financially capable of, go and do it man.

23 is still young and it's not about being the best, it's about being the best you can be. Have fun, work at it and push yourself.

Is it to late to be a pro.. almost certainly. 

Can you enjoy riding with your buddies for 25 years

100%

23 isn't too late to get into anything in life, my friend. Take it from someone who worried about the exact same thing around that age and is now 56 and ready to try new things. Just be the best racer you can be.

started riding on a track at 30, had my first race at 33 years old. Get on track and start riding consistently youll love it

Did my first race at 25, be humble, hold your line and don't ride over your head. I'm still super slow but I have fun out there, and you will too!

Make sure you are doing plenty of practice days as well, its really easy to enforce bad habits while racing as its hard to concentrate on your form and technique so want to balance it out with days where you are doing some solid practice.  

Focus on being solid and smooth and the speed will come on its own.

The 30+ C class is the biggest class at our local off-road races, I think its the perfect age where guys are still in good shape and have enough disposable income to do some racing! 

Ride a 250F. Don't listen to anyone that tells you to get a 450.

23 years old? You have years and years left in you to ride and race. I rode a lot of MX in my 20's, but never raced. I didn't start racing until I was 30. I'm 42 now and still ride and race. I'll do this as long as my body holds up. To be honest, I'm a far better rider at 42 then I was in my 20's. My body disagrees on Monday morning, but nothing a few Ibuprofen can't take care of. 🤣

Never to late, I have seen riders in their 50's start riding tracks and get into racing. It may be frustrating at first, just getting used to the way things are done but you will figure it out and enjoy.  Dont get frustrated when things do go well, its all for fun.

In hindsight, Id call 23 one of my Prime years. Seemed like maybe 25 started a turn off of peak for me. 

What Id do to be 23 again. 

Beat me to it. I'd love to be 23 again. OP is in his prime. Go get after it!

Never to late, There is a class for everyone at almost every track.

Seat time, seat time, seat time.

Don't rule out racing Hare Scrambles.  2 hour races are a good way to hone skills and build stamina.

Hell no, it’s not too late! One of the great things about Moto is…. You could end up being a top racer in a very short time… nobody knows who is a natural and who is not, until you try…get some quality lessons and put the work in…. Before you know it, you will be going to Loretta Lynn’s in your new 500k motorhome and posting selfie photos of yourself biting down on some cheap cardboard plaque.

Agreed, and honestly depending on your area harescrambles and enduros may be more realistic way to race.

Here in Central NC I can race a couple of hare scramble series and race almost every weekend this time of year, MX is much less.  

I race in a vet (+30) club and we get new guys every year in the beginner class... REAL beginners to dirtbikes and motocross. They have a blast.

23 is nothin'.

Never to late.

Don't be afraid to get last. Get to the gate and race. You learn so much your 1st few races its insane.

One thing you can do since you're just starting out. See if your local track has practice day the day before the race. do as many of those as you can before you show back up someday on a Sunday, You'll be ready.

It sounds like everybody is in agreement - it's never too late. Go to a race and watch how it's done, then bring your bike the next race weekend and line up! 

23 isn’t too late for anything. I have worked with guys that started boxing at 25 and were world champions at 35. Of course, that’s a different level of commitment and they had slightly different motivations. 
 

but to have fun racing with buddies? You are just fine!

I found most of the guys I was racing as kids started to get over it by the time we were all getting vehicles at 16-18. Now at 24-26 they all want to get back into it. 23 you’ll be able to always have a goal of improvement and never get over it