frame_mol -max -1,3 ref_frames.dat temp.alc temp2.alc
rotate_mol -r=rotz temp2.alc temp.alc
The "frame_mol" step already sets the view based on reference frames of bps #1 and #3. If you have several structures processed the same way, they should be directly comparable. So I do not think the rotate_mol step is necessary.
If you insist on "angle of rotation about z-axis", then you must define your z-axis as a vector in three-dimensional space. After "frame_mol", the 'global z-axis' is [0, 0, 1].
If I understand you correctly, you want to align your structure to the "global linear helical axis". The proper section for your sample structure is:
Global linear helical axis defined by equivalent C1' and RN9/YN1 atom pairs
Deviation from regular linear helix: 3.12(0.70)
Helix: -0.056 -0.258 -0.965
HETATM 9998 XS X X 999 -0.945 1.793 2.483
HETATM 9999 XE X X 999 -1.305 0.132 -3.733
Average and standard deviation of helix radius:
P: 8.44(0.65), O4': 7.14(0.31), C1': 6.19(0.34)
Note you need to set "std_curved" (in file misc_3dna.par or via command-line e.g. "-std_curved=0.8") to larger than the default 0.6 to see the output of this section for the sample structure. You may also need to specify x- and y-axis.
I am in the process of upgrading 3DNA to v2.2 (with a user manual!), the final version in the v2.x series. If you are willing to work together, I will consider your use case as an example to implement a feature that works for you.
HTH,
Xiang-Jun