[STS-1440] Possibility to attach source jars via ivy Created: 04/Jan/11  Updated: 12/Oct/11  Resolved: 24/Aug/11

Status: Resolved
Project: Spring Tool Suite
Component/s: GRAILS
Affects Version/s: None
Fix Version/s: 2.8.0.M2

Type: Improvement Priority: Major
Reporter: Konstantinos Kostarellis Assignee: Kris De Volder
Resolution: Complete Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

kubuntu 10.04, grails 1.3.6, jvm 1.6.22



 Description   

It would be great if STS could provide some functionality to download and attach the source jars of dependencies, retrieved though ivy - similar to what m2eclipse can do with maven.

STORY:
grails-plugins often provide additional libraries which are retrieved by ivy. it would be great to have some functionality to attach also the source jars automatically.

m2eclipse provides similar functionalities by right-click / download sources + download javadoc.

STATUS QUO:
I have to retrieve the source artifacts (which I usually do through some dummy-maven-project) and attach every source jar i need manually.

SITENOTE:
In my actual setup (STS 2.5.1) a restart is also needed to make the attached source work. Don't know if its just my setup or a bug. If someone can commit this, i can post an issue-report for this.



 Comments   
Comment by Kris De Volder [ 05/Jan/11 ]

Hi Konstantinos,

I agree, having to manually configure / download source jars is a pain. Ideally we should have these automatically added to the grails classpath container dependencies as much as possible.

I will look into this in the near future.

What would be a really big help for me is:

  • a concrete example (i.e. a Grails project that demonstrates the problem of a "missing jar")
  • some more details on where/how you would get the "missing" source-jar(s) in that example project

If I understand how/where to get source jars for the dependencies, than I think it shouldn't be
too hard to make sure the classpath container entries get automatically configured with source
jars attached.

Could you help me out?

Kris

Comment by Konstantinos Kostarellis [ 06/Jan/11 ]

Hi Kris,

every new grails projekt has jars on the classpath without the corresponding source-jars. (e.q. hibernate-core)
only the grails-* libs have the source-jars attached by default. If you install more plugins like searchable or spring-security-core etc. there are more libs attached to your classpath (no source-jars attached).
The jars are obviously handled by apache ivy. I was in a project before, where a former college had written an ant-task that would download the source jars of the dependencies. Similar to what m2eclipse can do for you.
He briefly mentioned online that he used the maven ant taks and pointed me to http://maven.apache.org/ant-tasks/examples/dependencies.html which I don't know if its right. I guess I will have to look for that old project and look into it to see how it was achieved.

Sorry for the pour help at the moment. I will try to investigate for some more infos.
Another source might be the m2eclipse plugin ... as they obviously achieved similar functionalities.

Comment by Kris De Volder [ 07/Jan/11 ]

Thanks Konstantinos,

I appreciate your pointing me in the right direction. I did a bit of googling myself on stuff like "source jars", "source attachements" and "ivy" but so far nothing too helpful has turned up.

I'll keep digging and figure something out somehow .

If you find more info you think could be helpful, please let me know (that script you mentioned may be helpful, if you could find it).

Kris

Comment by Konstantinos Kostarellis [ 07/Jan/11 ]

it appears that old project i thought of, was not managed by ivy but using pure ant and the maven ant-task to pull the dependencies of some pom.xml. So the source-jars where downloaded as it is described in http://maven.apache.org/ant-tasks/examples/dependencies.html

<copy todir="lib/src">
<fileset refid="sources.dependency.fileset" />
<mapper classpathref="maven-ant-tasks.classpath"
classname="org.apache.maven.artifact.ant.VersionMapper"
from="$

{dependency.versions}

" to="flatten" />
</copy>

But this approach doesn't seem fitting here.
Maybe somebody on the grails user-mailinglist or the apache ivy - mailinglist has better ideas/infos.

Comment by Konstantinos Kostarellis [ 08/Jan/11 ]

I guess I found something helpful.

As discussed on the grails-user mailing list
http://grails.1312388.n4.nabble.com/Grails-dependency-sources-td2956849.html
there is a grails plugin that offers a script, which offers the needed functionality.

http://www.grails.org/plugin/eclipse-scripts

If a similar functionality is integrated into STS, i can see how it makes STS more prevalent

I guess an option in the context menu offering the functionality like m2eclipse does, could be the right spot. Maybe you got a even better idea about where to place it.

Comment by Kris De Volder [ 10/Jan/11 ]

Thanks Konstantinos, that is very relevant and useful information. From what I've read there, there's also a related issue in Grails itself dealing with the source code of dependencies.

http://jira.codehaus.org/browse/GRAILS-5611

So it is probably a good idea to synch up with Grails people. If Grails would provide a way to get the sources, it would be much easier for STS to provide easy access by correctly setting up the classpath container with those sources.

But I may be able to do something with that plugin you found also.

Comment by Lari Hotari [ 02/Feb/11 ]

I've updated the Grails plugin "eclipse-scripts" to support STS Grails dependency management.

Quick example (for Grails project imported to STS):

grails install-plugin eclipse-scripts 1.0.3
grails compile
grails download-sources-and-javadocs
# run twice to download everything
grails download-sources-and-javadocs
grails sts-link-sources-and-javadocs
# now refresh your workspace in STS and restart it

The script modifies the .settings/com.springsource.sts.grails.core.prefs file.

Comment by Kris De Volder [ 02/Feb/11 ]

Lari,

Thanks... However, I'm not entirely thrilled at the prospect of a plugin modifying those settings files. Though the format may be stable for a long time, it is really an internal format and very much subject to change... (Also when they get rewritten by STS, I'm not sure the settings will persist!).

Anyway, since I don't have time to look into fixing this issue right away. I guess this does provide a valuable service to users right now... So thanks on behalf of our users

When I get round to working on this, I'd like to be able to use your plugin to download the sources / javadocs, but the final step of taking the info and downloads from the plugin and configuring them into the Eclipse / STS classpath and settings files is really something I'd prefer to handle inside of STS. That way we can give better guarantees it won't break when we make change the configuration files.

Kris

Comment by Lari Hotari [ 02/Feb/11 ]

Yes, this plugin is a hack. It's easy to modify the plugin when STS changes.

I just release version 1.0.4 (a few minutes ago) of eclipse-scripts . It downloads sources and javadocs also for transitive dependencies.

These steps are now the minimum to get sources and javadocs attached to a Grails project:

grails install-plugin eclipse-scripts 1.0.4
grails compile
grails download-sources-and-javadocs
grails sts-link-sources-and-javadocs

The DownloadSourcesAndJavadocs.groovy script is quite a hack. Take a look at the source code and you'll find out the tricks it does.

Comment by Kris De Volder [ 02/Feb/11 ]

Thanks for that source file link. Very interesting

I'm not just worried about changes in the format. I'm also worried about changes that happen when preferences change for a project. The file may be overwritten / destroyed by STS at any time while a user is working (STS is assuming it 'owns' that file).

But I like I said, since we don't have anything better now, this is actually useful

I'm hoping you'd be willing to work with me a bit when I get to this issue. Maybe your command / scripts could write some info about the downloaded dependencies in a file other than the eclipse settings (preferably I'd just pass the location of that file as an argument to the command or something like that) and I can wire something up in STS that reads and properly configures the project's class path with that info.

Kris

Comment by Lari Hotari [ 02/Feb/11 ]

StsLinkSourcesAndJavadocs.groovy is the other script. It just navigates the .ivy2/cache directory to find sources and javadocs (used after downloading).

One reason for the complexity of the DownloadSourcesAndJavadocs.groovy script is this Grails Jira issue: https://jira.codehaus.org/browse/GRAILS-6700 . Grails creates invalid Ivy xml files (ivy poms?) to .ivy2/cache directory for all of the jar files that come with Grails. Source code downloading doesn't work for invalid ivy xml files. I had to use a separate ivySettings.defaultCache directory because of this and copy *.jar files to/from this separate directory. Ivy xml files get loaded from repositories for all files (so the invalid ivy xml files don't get used).

It would be possible to change the output of the script easily. The problem is that these scripts are basicly hacks that I've just got to work (for my purposes) and they haven't been designed and well thought out.

What kind of file format or interface would you prefer? (we can make a proof-of-concept iteratively)

btw. In STS internals I'd prefer a settings file that doesn't contain any other information besides source code and javadoc locations. The reason is that in team development I'd like to ignore this file from SCM (svn/cvs/git/...) fully.
Supporting classpath / linked resource directory variables is optional for me. Even in Eclipse Javadoc urls don't support variables (variables are supported for sources).
Could you suggest separting source code and javadoc location information to a separate file in STS?
Currently there is some other preferences stored in the same file (.settings/com.springsource.sts.grails.core.prefs).

Comment by Kris De Volder [ 03/Feb/11 ]

Lari,

STS internally has a few hacks of its own to try and pry information out of Grails e.g. about dependencies. We wish there was a nice API to just ask Grails for this info, but unfortunately there isn't. This code/hacks are some of the most fragile pieces of STS/Grails tooling and cause us (me mostly a lot of headache.

We store this information in a file that is outside the project, in a place somewhere in the workspace metadata. Since it is outside the project, it won't be versioned (which is as it should be, as you point out).

We use this info to calculate the contents of the Grails Class Path container (i.e. the stuff you see under 'Grails Dependencies').

I tried a bit yesterday to play with your scripts. To be honest, I was a bit put-off by the amount of error messages and the very long time it took to execute the script. So, I'm not so sure now if would want to hookup the script directly from the STS UI.

However, if we were going to hook it up, the way I'd see it working would be that STS have a menu somewhere 'Download Source Attachements' or something like that. If the user selects it, it would execute the necessary grails commands. We'd pass the location of the file to you. We'd agree on some format (it needn't be complex, just a list of entries associating binary jars to their corresponding source / javadoc jars.

STS would read the file when the commands is completed and use the info in it to update the classpath container with source / javadoc attachements.

Kris

Comment by Kris De Volder [ 08/Feb/11 ]

Raising priority. A duplicate request was just raised.
https://issuetracker.springsource.com/browse/STS-1562

Comment by Eric Sirianni [ 04/May/11 ]

I'm a bit surprised that STS does not already have this feature. IvyDE and m2eclipse already have the source JAR attachment feature.

I'm not sure what STS is using behind the scenes to implement the "Grails Dependencies" classpath container. Since grails uses Ivy, I assumed you were using IvyDE.

If not, at least you might want to have a look at how IvyDE implements the source JAR attachment.

Comment by Kris De Volder [ 22/Aug/11 ]

Grails 2.0 will make this easier for us to support:
http://jira.grails.org/browse/GRAILS-7916

Comment by Kris De Volder [ 24/Aug/11 ]

I've already committed something that works. It only works for Grails 2.0.M2 since it makes use of the new grails refresh-dependencies command.

Also, presently the functionality won't work if the Eclipse workspace path contains a space. This is, unfortunately, because Grails argument parsing doesn't handle escape sequences so there is no way to pass a path with a space in it to Grails.

I have no intention of trying to make this work for Grails 1.3.X since that will be very hard. But I will try to resolve the space-path problem before closing this issue.

Comment by Kris De Volder [ 24/Aug/11 ]

Note: if someone wants to try this out... you will need to get update after the next nightly snapshot STS build from:

http://dist.springsource.com/snapshot/TOOLS/nightly/e3.7/

Then you'll also need to use grails snapshot distro from here:
http://hudson.grails.org/job/grails_core_2.0.x/lastSuccessfulBuild/artifact/build/distributions/

Comment by Kris De Volder [ 24/Aug/11 ]

Committed something to handle spaces in path.

It works, assuming grails pull request
https://github.com/grails/grails-core/pull/99

is applied to Grails.

Comment by Davide Cavestro [ 16/Sep/11 ]

Using |Grails tools|Download Source Jars| works for me, but I must say I have no spaces in path... I simply hate them too much
Just given a try with an ad-hoc project, cause I have no Grails 2 real world projects till now (hope to start some in the near future).

Comment by Kris De Volder [ 16/Sep/11 ]

FYI. This functionality should work with upcoming STS 2.8.M2 and already released Grails 2.0.0.M2.

Kris

Comment by Konstantinos Kostarellis [ 11/Oct/11 ]

Worked great on a few small projects (first steps), but failed on the first plugin-project.
This seems to be a grails problem, thou. I just raised an issue on that -> http://jira.grails.org/browse/GRAILS-8157

I could think of another improvement, similar to the way m2e handles it. When opening a class of a
dependency that has not source attached, m2e starts downloading it in the background
and refreshes the editor window when its done. Don't know if this is possible to implement.
I could file an jira-improvment-issue on this (minor), if this is some desired functionality.

So far I'm very impressed with the improvements. Nice work!

Comment by Kris De Volder [ 11/Oct/11 ]

Hi Konstantinos,

> When opening a class of a
> dependency that has not source attached, m2e starts downloading it in the background
> and refreshes the editor window when its done. Don't know if this is possible to implement.

I'd have to look into it to figure out how hard it would be. But given m2e does something like it probably means it is doable.

Sure, open a Jira issue, can't promise this will actually get implemented, however its good to have a Jira issue so people can express interest, comment, vote etc. The votes help us to determine what issues are more important to the users.

Also, thanks for raising the Grails Jira issue on the plugin problem.

Comment by Konstantinos Kostarellis [ 12/Oct/11 ]

Hi Kris,

just stumbled over another glitch regarding the attached source libs.
STS source references for the libs grails-datastore-core-1.0.0.M10.jar and
grails-datastore-gorm-1.0.0.M10.jar are pointing towards $GRAILS_HOME/src/,
but those source jars ain't packaged into the gails-2.0.0.M2.zip

Using grails-2.0.0.M2 and STS 2.8.0.SNAPSHOT (Build Id: 201110120725)

A simple workaround is to grab the source-jars from http://repo.grails.org/grails/core/org/grails/grails-datastore-core/ and place them in $GRAILS_HOME/src/

Comment by Kris De Volder [ 12/Oct/11 ]

Could you raise that as a separate issue? I can take a look, maybe we are handling this the wrong way. Could be they are handled correctly by the 'download sources' grails 2.0 feature but we (STS) are mistakenly pointing to the grails distro's source dir.

Generated at Sat Nov 25 11:06:12 UTC 2017 using JIRA 6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5.