If the fixed brackets that receive the posts (top and bottom) aren't plumb to each other this will happen and frankly drive you crazy trying to adjust them.
Remove the first door completely.
Get a plumb bob, and hang it from the top bracket - if the lower bracket (which receives the lower pin/post from the fixed side of the bi-fold door) isn't exactly straight and plumb beneath it - then you'll need to relocate either the upper track (if present) location of the upper bracket, or re-locate the position of the lower bracket.
Don't rely on the side wall finish being either straight, level or plumb - or your floor for that matter.
If these bifolds do not have a top rail/track that the upper brackets are set into, please advise. If you do have a floor bracket/track also advise (assuming top track, and no bottom track here).
Next you'll have to determine if both lower brackets are set LEVEL and straight to each other, and shim one or the other as necessary to the floor side - relying on spring tension of the adjustment post to level will just lead to frustration.
Expect that your closet opening is not straight, plumb or true - and neither outside wall, either to itself, nor to each other.
If your top track not installed perfectly straight, level, true, and twisted, warped, etc. in any way - you will also have difficulty in aligning meeting bi-fold doors.
your gap to the walls on either side of the doors may well need to be NOT uniform - to get the doors themselves to hang straight, level and plumb - and in alignment to each other. Creative trimming and moulding can be used to "case out" this "zone" sometimes first installing some door stop trim then casing overtop on the pin/fixed sides of the closet opening)or some wall finishing work can be done to square off and true-out the closet opening.
Bottom line: you need a plumb bob, string, measuring tape, and a short and long true level to investigate your closet opening sans (without) the doors in place to begin with.